On the Convergence of Federated Optimization in Heterogeneous Networks

By Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, Virginia Smith. 2018

375 citations recorded by the source corpus.

Subfields: Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, methodology): “and ‖ŵ i − w̃‖ ≤ 1 μ ‖∇Fi(w̃)‖, (49) where θ denotes a θ solution of minw Ji(w; w̃ ) [18].”
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “[22] Tian Li, Anit Kumar Sahu, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith.”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “To tackle the challenge of statistical heterogeneity, distributed optimization methods such as Adaptive Federated Optimizer [2], FedNova [3], FedProx [4], and FedMA [5] have been proposed.”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “…with decentralized training: notably, high communication cost (Tang et al., 2018a; Wang et al., 2019; Koloskova et al., 2019), a need for time-varying topologies (Nedi´c & Olshevsky, 2014; Assran et al., 2019) and data-heterogeneity (Li et al., 2018; Karimireddy et al., 2019; Li et al., 2020a;b).”
Personalized Federated Learning: A Meta-Learning Approach (2020, background): “More related to our paper, there are several works that study statistical heterogeneity of users’ data points in FL (Zhao et al., 2018; Sahu et al., 2018; Karimireddy et al., 2019; Haddadpour and Mahdavi, 2019; Khaled et al., 2019; Li et al., 2019), but they do not attempt to ﬁnd a personalized…”
Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [35, 51, 56, 59] — see [33] for a survey.”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, methodology): “We can also assume that due to the availability criterion described before, such available devices are randomly and uniformly distributed over the network [Sahu et al., 2018].”
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (2019, background): “Sahu et al. [2018] introduce a proximal term to client objectives and prove convergence guarantees.”
First Analysis of Local GD on Heterogeneous Data (2019, background): “We note that an attempt to get more general convergence statement has been made in [13], but sadly their guarantee is strictly worse than that of minibatch Stochastic Gradient Descent (SGD), making their theoretical contribution smaller.”
Federated Visual Classification with Real-World Data Distribution (2020, background): “[25] introduce a proximal term to client objectives for bounded variance.”
Federated Learning with Matched Averaging (2020, methodology): “, 2017) and FedProx (Sahu et al., 2018); analyze the performance under the growing number of clients and visualize the matching behavior of FedMA to study its interpretability.”
On Safeguarding Privacy and Security in the Framework of Federated Learning (2019, background): “As pointed out in [13], the theoretical convergence guarantees have not been fully explored in the federated average learning, even although recent works can provide approximate convergence guarantee to some extent.”
Federated Class-Incremental Learning (2022, background): “[38] introduces a proximal term to help local model approximate the global ones.”
Federated Meta-Learning with Fast Convergence and Efficient Communication (2018, methodology): “To handle the statistical and systematic challenges in the ferderated setting, many optimization methods have been proposed [14, 20, 13], which have signiﬁcantly improve the convergence performance and system overhead.”
On the Convergence of Local Descent Methods in Federated Learning (2019, methodology): “In contrast to [20] and [28] that focus on analyzing the convergence of federated optimization for strongly-convex functions under restricted assumptions as shown in Table 1, our analysis is for general non-convex problems yet under weaker assumptions (e.g., removing bounded gradient assumption).”
An Efficient Framework for Clustered Federated Learning (2020, methodology): “personal devices are usually non-i.i.d. Several formulations and solutions have been proposed to tackle this problem. A line of research focuses on learning a single global model from non-i.i.d. data [48, 33, 20, 36, 23, 29]. Other lines of research focus more on learning personalized models [37, 35”
Convergence Time Optimization for Federated Learning Over Wireless Networks (2020, neutral): “distribution [20] or statistical heterogeneity [21].”
Federated Learning on Non-IID Data: A Survey (2021, background): “Since the local data distributions are different from the global data distribution, the averaged local model parameters may also be far away from the global model parameters [166], particularly when the number of epochs for local updates is large [62, 110, 130].”
FetchSGD: Communication-Efficient Federated Learning with Sketching (2020, neutral): “Sahu et al. (2018) propose constraining the local gradient update steps in FedAvg by penalizing the L2 distance between local models and the current global model.”
Decentralized Federated Averaging (2021, background): “The convergence analysis of FedAvg on heterogeneous data is discussed in [14], [15], [16].”