Federated learning with hierarchical clustering of local updates to improve training on non-IID data

By Christopher Briggs, Zhong Fan, Péter András. 2020

767 citations recorded by the source corpus.

Federated learning (FL) is a well established method for performing machine learning tasks over massively distributed data. However in settings where data is distributed in a non-iid (not independent and identically distributed) fashion - as is typical in real world situations - the joint model produced by FL suffers in terms of test set accuracy and/or communication costs compared to training on iid data. We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. In this work we present a modification to FL by introducing a hierarchical clustering step (FL+HC) to separate clusters of clients by the similarity of their local updates to the global joint model. Once separated, the clusters are trained independently and in parallel on specialised models. We present a robust empirical analysis of the hyperparameters for FL+HC for several iid and non-iid settings. We show how FL+HC allows model training to converge in fewer communication rounds (significantly so under some non-iid settings) compared to FL without clustering. Additionally, FL+HC allows for a greater percentage of clients to reach a target accuracy compared to standard FL. Finally we make suggestions for good default hyperparameters to promote superior performing specialised models without modifying the the underlying federated learning communication protocol.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Dacheng Tao
3. Murali Annavaram
4. Chaoyang He
5. Yang Liu
6. Guodong Long
7. Jing Jiang
8. Chengqi Zhang

In Computer Science Applications

1. Li-zhen Cui
2. Dacheng Tao
3. Peilin Zhao
4. Azzam Mourad
5. Murali Annavaram
6. Ramesh Raskar
7. Bo Du
8. Qiang Yang

Citations in the corpus

• FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “As demonstrated in Figure 3, FL has various topology definitions, such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], and Hierarchical FL [47, 48, 49, 50, 51, 52].”
• FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, unknown): “[3, 30] clustered similar clients to match the data distribution within a cluster and learned separate models for each cluster without inter-cluster federation.”
• From distributed machine learning to federated learning: a survey (2021, background): “In addition, by well-clustering the computing resources to groups, the hierarchical architecture is also exploited to address unbalanced data distributed among multiple computing resources [19,126], or to address data privacy [172].”
• Federated Learning for Open Banking (2021, background): “Moreover, [5] discussed using different measurements. where the W i and W j are the model parameters of participants i ad j respectively.”
• Towards Personalized Federated Learning (2021, background): “However, they incur high computation and communication costs that limit practical feasibility for largescale settings [66], [67].”
• Personalized Federated Learning with First Order Model Optimization (2020, background): “FedProx (Li et al., 2020) et al., 2020; Ghosh et al., 2020; Briggs et al., 2020; Mansour et al., 2020) assumes that inherent partitions or data distributions exist behind clients’ local data, and aim to cluster these partitions to federate within each cluster.”
• Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, methodology): “In [64], the authors propose to group the clients based on the similarity of their local updates to the global joint model using a hierarchical clustering mechanism.”
• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges (2022, background): “Furthermore, since nodes within a cluster share similar data distributions, individually trained cluster models may be less generic and robust than models exposed to global data distributions [57].”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “[15] aim to decrease the number of communication rounds required to reach convergence while improving the test accuracy.”