Adaptive Federated Optimization
By Sashank J. Reddi, Zachary B. Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, H. Brendan McMahan. 2020
2,039 citations recorded by the source corpus.
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Due to the heterogeneity of the client datasets, standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general nonconvex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.
Subfields: Artificial Intelligence, Optimization & Training, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Martin Jaggi
- Qiang Yang
- Peter Richtárik
- Jakub Konecný
- Yoshua Bengio
- Dacheng Tao
- Percy Liang
- Sergey Levine
In Optimization & Training
- Jakub Konecný
- Peter Richtárik
- Martin Jaggi
- Jianyu Wang
- Sebastian U. Stich
- Gauri Joshi
- Aryan Mokhtari
- Mehrdad Mahdavi
In Federated & Distributed Learning
- Jianyu Wang
- Gauri Joshi
- Qiang Yang
- Virginia Smith
- Martin Jaggi
- Jakub Konecný
- Mehrdad Mahdavi
- Bingsheng He
Citations in the corpus
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, methodology): “Other techniques like acceleration, recently appear in preprints (Hsu et al., 2019; 2020; Reddi et al., 2020).”
- Practical and Private (Deep) Learning without Sampling or Shuffling (2021, methodology): “For StackOverflow, we use the one-layer LSTM network described in (Reddi et al., 2020).”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, methodology): “Models For EMNIST, we train a small convolutional net similar to the model defined in Reddi et al. (2020) but with model size d slightly under 220 parameters to reduce padding from the Walsh-Hadamard transform.”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “CIFAR100 used the usual pre-processing (normalization and centering), and data augmentation (random crop and horizontal flipping) following [21].”
- Flower: A Friendly Federated Learning Research Framework (2020, methodology): “A family of server-side optimizations that include FedAdagrad, FedYogi, and FedAdam as described in Reddi et al. (2021).”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “In this setting, we consider three data sets: MNIST [132], Federated EMINST [4], and Synthetic ( α , β ) [3], with the logistic regression model as the benchmark.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “The analysis relies on Assumptions 1 and 2 used in the standard analysis of SGD [51] and Assumption 3 commonly used in the federated optimization literature [38, 12, 13, 20, 40, 2] to capture the dissimilarities of local objectives.”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “However, these versions instead perform averaging on the so-called “model delta” (see (Reddi et al., 2020)), in which the server model is updated via xt+1 = xt − η M ∑ i∈It (xt − xit) = xt − η M ∑ i∈It ( xt − ( xt − γ K∑ k=1 gik )) = xt − ηγ M ∑ i∈It K∑ k=1 gik.”
- Personalized Federated Learning with Moreau Envelopes (2020, methodology): “According to Corollary 2, we are able to obtain the sublinear speedup O ( 1/(TRN) ) , while most of existing convergence analysis for nonconvex FL can only achieve a sublinear speed-up of O ( 1/ √ TRN ) [3, 6, 49].”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- Differentially Private Learning with Adaptive Clipping (2019, positive): “Combining our results with the lessons taken from (McMa-han et al., 2018) and (Reddi et al., 2020), the following strategy emerges for training a high-performing model with user-level differential privacy.”
- Distilled One-Shot Federated Learning (2020, background): “The work in [26] proposes several federated version of optimizers, FedAdaGrad, FedAdam and FedYogi, to provide better convergence for non-IID data.”
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “This is confirmed by both this study and other CNN-based FL works [26, 27]; 2.”
- Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning (2021, methodology): “In order for this paper to be self-contained, we restate the proof of Lemma 4 in (Reddi et al., 2020) here.”
- Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data (2020, background): “Following Lemma 4 in Reddi et al. (2020), we can derive the corresponding drift bound for our problem that for γt ≤ 1 8LK N ∑”
- FedScale: Benchmarking Model and System Performance of Federated Learning at Scale (2021, positive): “2 Background Existing efforts optimize for various goals of practical FL To tackle heterogeneous client data, FedProx [36], FedYogi [43] and Scaffold [32] introduce adaptive client/server optimizations that use control variates to correct for the ‘drift’ in model updates.”
- Federated Learning with Compression: Unified Analysis and Sharp Guarantees (2020, background): “+ mGq Rτγ2 (q + 1) ) , then by letting γ ≥ m we improve the convergence rate of [19] and [36] with tuned global and local learning rates, showing that we can archive the error with R = Θ ( (1 + q) −1 ) and τ = Θ ( 1 m ) , which matches the communication and computational complexity of [19] and [26],”
- On Large-Cohort Training for Federated Learning (2021, methodology): “This intuition is reflected in many convergence analyses of FL methods (Khaled et al., 2019; Karimireddy et al., 2020b; Khaled et al., 2020; Reddi et al., 2021; Yang et al., 2021), which generally show that asymptotic convergence rates improve as the cohort size increases.”