Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

By Valentina Zantedeschi, Aurélien Bellet, Marc Tommasi. 2019

90 citations recorded by the source corpus.

We consider the fully decentralized machine learning scenario where many users with personal datasets collaborate to learn models through local peer-to-peer exchanges , without a central coordinator. We propose to train personalized models that leverage a collaboration graph describing the relationships between the users' personal tasks, which we learn jointly with the models. Our fully decentralized optimization procedure alternates between training nonlinear models given the graph in a greedy boosting manner, and updating the collaboration graph (with controlled sparsity) given the models. Throughout the process, users exchange messages only with a small number of peers (their direct neighbors in the graph and a few random users), ensuring that the procedure naturally scales to large numbers of users. We analyze the convergence rate, memory and communication complexity of our approach, and demonstrate its benefits compared to competing techniques on synthetic and real datasets.

Subfields: Artificial Intelligence, Information Systems, Graph Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hai Helen Li
2. Yiran Chen
3. Giovanni Neglia
4. Othmane Marfoq
5. Richard Vidal
6. Laetitia Kameni
7. Aurélien Bellet
8. Yutao Huang

In Information Systems

1. Jian Pei
2. Aurélien Bellet
3. Jiankun Hu
4. Binghui Wang
5. Yanming Zhu
6. Xuefei Yin
7. Neil Zhenqiang Gong
8. Cynthia Rudin

In Graph Learning

1. Jian Pei
2. Binghui Wang
3. Yiran Chen

In Federated & Distributed Learning

1. Othmane Marfoq
2. Richard Vidal
3. Giovanni Neglia
4. Laetitia Kameni
5. Yutao Huang
6. Jian Pei
7. Jiangchuan Liu
8. Aurélien Bellet

Citations in the corpus

• Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “Multi-Task Learning (MTL) has recently emerged as an alternative approach to learn personalized models in the federated setting and allows for more nuanced relations among clients’ models (Smith et al., 2017; Vanhaesebrouck et al., 2017; Zantedeschi et al., 2020; Hanzely & Richtárik, 2020; Dinh et a”
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets (2020, background): “Existing works achieve personalization via meta-learning [4–6], multi-task learning [7, 8], transfer learning [9, 10], etc.”
• Personalized Federated Learning: An Attentive Collaboration Approach (2020, background): “Distributed multi-task learning [33, 34, 35, 36] tried to conduct multi-task learning when the data for each task is separately owned by different clients.”
• A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “Compared to the client-server architecture, there is no central server [151, 214].”