Decentralized Collaborative Learning of Personalized Models over Networks

By Paul Vanhaesebrouck, Aurélien Bellet, Marc Tommasi. 2016

252 citations recorded by the source corpus.

We consider a set of learning agents in a col-laborative peer-to-peer network, where each agent learns a personalized model according to its own learning objective. The question addressed in this paper is: how can agents improve upon their locally trained model by communicating with other agents that have similar objectives? We introduce and analyze two asynchronous gossip algorithms running in a fully decentralized manner. Our first approach , inspired from label propagation, aims to smooth pre-trained local models over the network while accounting for the confidence that each agent has in its initial model. In our second approach, agents jointly learn and propagate their model by making iterative updates based on both their local dataset and the behavior of their neighbors. Our algorithm to optimize this challenging objective in a decentralized way is based on ADMM.

Subfields: Artificial Intelligence, Management Science and Operations Research, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Federated & Distributed Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “The setting of decentralized SGD also naturally lends itself to asynchronous algorithms in which each client becomes active independently at random times, removing the need for global synchronization and potentially improving scalability [110, 392, 54, 30, 267].”
Privacy Amplification by Decentralization (2020, background): “[37, 18, 52, 38, 51, 11, 45, 35] for recent work and [32] (Section 2.”
Personalized Federated Learning with Moreau Envelopes (2020, neutral): “[31] achieves personalization on each user in a fully decentralized network using asynchronous gossip algorithms with assumptions on network topology and similarity between users.”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “To further protect the data privacy and avoid the communication bottleneck, the decentralized architecture has been recently proposed [5, 6], where the centralized node has been removed, and each node only communicates with its neighbors (with mutual trust) by exchanging their local models.”
Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs (2019, methodology): “3, Dada-Learned and Perso-lin-Learned, which both make use of our alternating procedure, achieve the best performance.”
Personalized and Private Peer-to-Peer Machine Learning (2017, methodology): “We first conduct experiments on a linear classification task introduced by Vanhaesebrouck et al. (2017).”
Adaptive Personalized Federated Learning (2020, background): “Some of these works on the personalization of models in a decentralized setting can be found in Vanhaesebrouck et al. (2017); Almeida and Xavier (2018), where in addition to the optimization error, they have network constraints or peer-to-peer communication limitation Bellet et al. (2017);…”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Though large organizations and companies may afford to place a powerful, robust, and secure central server to carry out the training process, all types of sectors cannot adapt that [71].”
Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework (2020, background): “, IoT based personalized smart healthcare, smart home services and applications, fine-grained location-aware recommendation services, and on-premise intelligent video analytics), personalized learning has recently attracted great attention [5], [6].”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, background): “Distributed multi-task learning [33, 34, 35, 36] tried to conduct multi-task learning when the data for each task is separately owned by different clients.”
Distributed Multi-task Learning for Sensor Network (2017, background): “[16] proposed asynchronous algorithms for collaborative peer-to-peer network based on label propagation which jointly learned and propagated their model based on both their local dataset and the behavior of their neighbors.”
Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges (2022, methodology): “These models optimize metrics for each task and generate a corresponding output [91].”
Federated Multi-Task Learning under a Mixture of Distributions (2021, methodology): “For these reasons, they have attracted significant interest recently in the machine learning community [41, 63, 42, 62, 3, 51, 46, 31].”
Who started this rumor? Quantifying the natural differential privacy guarantees of gossip protocols (2019, unknown): “They are a basic building block to propagate and aggregate information in distributed databases [5, 6] and social networks [7, 8], to model the spread of infectious diseases [9], as well as to train machine learning models on distributed datasets [10, 11, 12, 13].”
Throughput-Optimal Topology Design for Cross-Silo Federated Learning (2020, background): “[100] Paul Vanhaesebrouck, Aurélien Bellet, and Marc Tommasi.”
Personalized Federated Learning With Differential Privacy (2020, background): “Although some private and personalized learning scheme with a fully decentralized architecture like [28] has been proposed, the architecture with central servers will be more efficient especially for applications that are large scale and require high”
Lower Bounds and Optimal Algorithms for Personalized Federated Learning (2020, background): “There are multiple strategies in the literature for incorporating e personalization into FL: multi-task learning [49, 48, 14], transfer learning [54, 23], variational inference [8], mixing of local and global models [41, 19, 31, 10] and others [13].”
A Framework for Evaluating Gradient Leakage Attacks in Federated Learning (2020, background): “Federated learning enables the training of a high-quality ML model in a decentralized manner over a network of devices with unreliable and intermittent network connections [37,26,40,19,5,44].”