Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

By Felix Sattler, K. Müller, Wojciech Samek. 2019

1,446 citations recorded by the source corpus.

Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients’ data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Guodong Long
3. Jing Jiang
4. Philip S. Yu
5. Qiang Yang
6. Tianyi Zhou
7. Chengqi Zhang
8. Jian Liang

In Computer Science Applications

1. Shiqing Liu
2. Jinjin Xu
3. Yaochu Jin
4. Hangyu Zhu
5. Tian Li
6. Virginia Smith
7. Li-zhen Cui
8. Zhong Fan

Citations in the corpus

• Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, methodology): “an additional k-means clustering in the average aggregation.2 Alternatively, one might try grouping the clients’ population sharing jointly trainable data distributions, as proposed by Sattler et al. [37] in the context of deep neural networks. An additional complication is constituted by the preser”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “A novel method called Clustered Federated Learning (CFL) [180] measures the similarity between the data distributions of different clients in terms of the cosine similarity between their gradient updates.”
• FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, methodology): “To tackle the statistical heterogeneity of data distributions, one straightforward solution is to maintain multiple global models for different local distributions, e.g., the works for clustered FL (Sattler, M¨uller, and Samek 2020).”
• Towards Personalized Federated Learning (2021, background): “In [Sattler et al., 2019], the authors proposed integrating hierarchical clustering into FL as a postprocessing step.”
• Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “Personalized FL has also been explored through clustering (e.g., Ghosh et al., 2020; Sattler et al., 2020; Muhammad et al., 2020), finetuning/transfer learning (Zhao et al.”
• Personalized Federated Learning with First Order Model Optimization (2020, background): “In contrast, prior work that personalized based on similarity to a client’s own model parameters (Mansour et al., 2020; Sattler et al., 2020) restricts this optimization to the local data distribution.”
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “4) personalized federated learning [19, 20, 21, 22], which aims to train personalized models for individual clients rather than a shared global model.”
• An Efficient Framework for Clustered Federated Learning (2020, methodology): “, the clustered Federated Learning [36, 26].”
• Federated learning with hierarchical clustering of local updates to improve training on non-IID data (2020, methodology): “[5] who develop a clustered federated learning (CFL) procedure.”
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey (2021, methodology): “In [86], the geometric properties of the FL loss surface are used to group nodes into clusters.”
• Federated Learning on Non-IID Data: A Survey (2021, methodology): “Before clustering the clients, some methods [7, 111] train and warm up the global model with the standard FedAvg algorithm.”
• Decentralized Federated Learning: A Survey and Perspective (2023, background): “A famous architecture is clustered FL that has been of interest to researchers [83], [125].”