An Efficient Framework for Clustered Federated Learning

By Avishek Ghosh, Jichan Chung, Dong Yin, Kannan Ramchandran. IEEE Transactions on Information Theory, 2020

1,254 citations recorded by the source corpus.

We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent. We analyze the convergence rate of this algorithm first in a linear model with squared loss and then for generic strongly convex and smooth loss functions. We show that in both settings, with good initialization, IFCA is guaranteed to converge, and discuss the optimality of the statistical error rate. In particular, for the linear model with two clusters, we can guarantee that our algorithm converges as long as the initialization is slightly better than random. When the clustering structure is ambiguous, we propose to train the models by combining IFCA with the weight sharing technique in multi-task learning. In the experiments, we show that our algorithm can succeed even if we relax the requirements on initialization with random initialization and multiple restarts. We also present experimental results showing that our algorithm is efficient in non-convex problems such as neural networks. We demonstrate the benefits of IFCA over the baselines on several clustered FL benchmarks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/06bc35120d2775bb4c26d6fd1ba68d1befcd84cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Philip S. Yu
3. Bo Du
4. Guodong Long
5. Qiang Yang
6. Jing Jiang
7. Mang Ye
8. H. Vincent Poor

Citations in the corpus

• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “The work in [79] addressed the FL problem, in which the users are distributed and partitioned into clusters.”
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “There are also heuristic approaches that divide clients into different clusters based on their learning tasks (objectives) and perform aggregation only within the cluster [33, 34, 22, 35].”
• Multi-center federated learning: clients clustering for better personalization (2020, background): “The user’s personal data will be stored locally to train the intelligent model, thus no personal data will be uploaded to the server.”
• Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “Personalized FL has also been explored through clustering (e.g., Ghosh et al., 2020; Sattler et al., 2020; Muhammad et al., 2020), finetuning/transfer learning (Zhao et al., 2018; Yu et al., 2020), meta-learning (Jiang et al., 2019; Chen et al., 2018; Khodak et al., 2019; Fallah et al., 2020; Li et…”
• FedScale: Benchmarking Model and System Performance of Federated Learning (2021, background): “, client clustering [12]) to obtain better training accuracy with fewer training rounds; (3) Privacy and security: developing reliable strategies (e.”
• FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, methodology): “The third solution is to provide multiple global models by clustering the local models (Mansour et al. 2020; Ghosh et al. 2020; Sattler, M¨uller, and Samek 2020) into multiple groups or clusters.”
• Towards Personalized Federated Learning (2021, methodology): “It can also be simulated by augmenting datasets via rotations [68].”
• Federated Learning for Open Banking (2021, methodology): “Then, [13] conducts convergence rate analysis in the same method.”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “The server-level methods require server participation, such as participating client selection [209, 225], or client clustering [60, 180, 206].”
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, methodology): “For example, FedAvg [1] and its variants [61, 62, 63, 64] can synchronize θ every round, and personalized FL methods can also adapt to this framework without much modification [27, 7, 65].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “IFCA [61] assumes that the parties are partitioned into clusters by the local objectives.”
• Decentralized Federated Learning: A Survey and Perspective (2023, background): “1) Client heterogeneity predominantly stems from three latent factors: 1) individual; 2) group; and 3) systemic heterogeneity [80], [83], [84], [85].”
• Personalized Federated Learning with First Order Model Optimization (2020, background): “FedProx (Li et al., 2020) et al., 2020; Ghosh et al., 2020; Briggs et al., 2020; Mansour et al., 2020) assumes that inherent partitions or data distributions exist behind clients’ local data, and aim to cluster these partitions to federate within each cluster.”