Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
By Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H. Vincent Poor. 2020
2,059 citations recorded by the source corpus.
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
Subfields: Artificial Intelligence, Computer Networks and Communications, Optimization & Training, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Networks and Communications
In Optimization & Training
- Jianyu Wang
- Aryan Mokhtari
- Gauri Joshi
- Yae Jee Cho
- Giovanni Neglia
- Othmane Marfoq
- Richard Vidal
- Yuanming Shi
In Federated & Distributed Learning
- Qiang Yang
- Murali Annavaram
- Chaoyang He
- Yang Liu
- Bingsheng He
- Qinbin Li
- Praneeth Vepakomma
- S. Avestimehr
Citations in the corpus
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “We will keep following the latest algorithm to be published at top-tier machine learning conferences, and will continuously add new FL algorithms such as Adaptive Federated Optimizer [2], FedNova [3], FedProx [4], and FedMA [5] in near future.”
- Distilled One-Shot Federated Learning (2020, background): “The issue of objective inconsistency was discussed in [36] and solved by the proposed FedNova, which averages the normalized local gradients.”
- FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction (2022, background): “With the highly skewed non-IID data in FL, the performance of FedAvg is significantly reduced [17, 23], which indicates that the FedAvg method that ignores local drift leads to the deviation of the global model.”
- Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “A normalized model aggregation method was also developed to ensure that the global model converges to the desired true model [142].”
- Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “Assumption 7 limits the level of dissimilarity of the different tasks, similarly to what is done in [66].”
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout (2021, methodology): “Until now, a lot of research effort has been invested with the sole goal of maximising the accuracy of the global model [49, 45, 42, 34, 67], while complementary mechanisms have been proposed to ensure privacy and robustness [7, 17, 51, 52, 30, 5].”
- No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data (2021, background): “[13] considers the setting where each client performs variable amounts of local works and proposes FedNova which normalizes the local updates before averaging.”
- Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, neutral): “To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30].”
- Federated Learning on Non-IID Data: A Survey (2021, methodology): “Since the imbalance level of data distributions can be easily adjusted by , some recent studies [80, 86, 130, 131] adopt this method to emulate different Non-IID cases in the real-world.”
- AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning (2021, methodology): “We compare AutoFL with two closely related prior works: FedNova [121] and FEDL [26].”
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “Substantial efforts have been made to address client heterogeneity in FL (Liang et al., 2019; Li et al., 2020b,a; Wang et al., 2020a; Zhang et al., 2021; Haddadpour et al., 2021; Yuan and Li, 2022; Alghunaim, 2023; Cheng et al., 2024), and develop other FL protocols involving variance reduction…”
- FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, methodology): “Moreover, functionality-based neural matching across local models (Wang et al. 2020a) can aggregate neurons with similar functionality regardless of the variance of the model architectures.”
- Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms (2020, background): “…i.i.d. setting [Yu et al., 2019, Karimireddy et al., 2019, Woodworth et al., 2020b] as well as explored various non-i.i.d. settings with different notions of heterogeneity [Zhao et al., 2018, Sahu et al., 2018, Hsieh et al., 2019, Li et al., 2019, Wang et al., 2020, Woodworth et al., 2020a].”