Communication-Efficient Learning of Deep Networks from Decentralized Data
By H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas. 2016
25,704 citations recorded by the source corpus.
Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.
Subfields: Artificial Intelligence, Transportation, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Dacheng Tao
- Quoc V. Le
- Sergey Levine
- Jianfeng Gao
- Ruslan Salakhutdinov
In Transportation
- Peter Richtárik
- Jakub Konecný
- Virginia Smith
- Qiang Yang
- Zachary B. Charles
- Ameet Talwalkar
- Aryan Mokhtari
- Medhi Bennis
In Federated & Distributed Learning
- Jianyu Wang
- Qiang Yang
- Gauri Joshi
- Peter Richtárik
- Virginia Smith
- Jakub Konecný
- Aryan Mokhtari
- Bingsheng He
Citations in the corpus
- Training Neural Networks with Fixed Sparse Masks (2021, background): “…assume all workers sample data i.i.d. from the same dataset and that all workers compute the same number of updates between each communication step, though our results could carry over to settings where workers use different datasets and varying amounts of updates (e.g. in Federated Learning [35]).”
- Advances and Open Problems in Federated Learning (2019, background): “[1]: “We term our approach Federated Learning, since the learning task is solved by a loose federation of participating devices (which we refer to as clients) which are coordinated by a central server.”
- Federated Optimization in Heterogeneous Networks (2018, methodology): “In Federated Averaging (FedAvg) (McMahan et al., 2017), the local surrogate of the global objective function at device k is Fk (·), and the local solver is stochastic gradient descent (SGD), with the same learning rate and number of local epochs used on each device.”
- How To Backdoor Federated Learning (2018, methodology): “The model is a 2-layer LSTM with 10 million parameters trained on a randomly chosen month (November 2017) from the public Reddit dataset, as in [23].”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “The master device could also randomly pick a subset of worker devices and send the global model to them; the picked worker devices update their local models and send them to the master device; and the master device aggregates the local models to obtain the new global model [39].”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, methodology): “We compare FedDF with SOTA FL methods, including 1) FEDAVG (McMahan et al., 2016), 2) FEDPROX (Li et al.”
- Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, background): “Federated learning (FL) is a recent advance in distributed ML in which data are acquired and processed locally at the client side, and then the updated ML parameters are transmitted to a central server for aggregation [6]–[8].”
- Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, positive): “Federated Averaging, introduced in [16], aims at reducing the necessary number of rounds of communication when training a federated model.”
- Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “This assumption was introduced and validated by (McMahan et al., 2017b), which showed that keeping the clipping norm bound, training rounds, and the scale of the noise added to the model update constant, increasing the report goal does not change the final model accuracy.”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, methodology): “, mobile devices or institutions) collaboratively train a model under the orchestration of a central server, while keeping training data decentralized (McMahan et al., 2017; Bonawitz et al., 2019a).”
- Differentially Private Learning Needs Better Features (or Much More Data) (2020, methodology): “DP has been considered for Federated Learning (Geyer et al., 2017; McMahan et al., 2018), but has also been found to significantly degrade performance in some settings (Yu et al., 2020).”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “All aggregation is weighted by the number of samples on the client as is standard [23, 27].”
- SplitNN-driven Vertical Partitioning (2020, methodology): “Other lines of work try to avoid these challenges (Liu et al. , 2020) following the same design principles as (McMahan et al. , 2017), and they propose Federated Stochastic Block Coordinate Descent (Fed-BCD).”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “• Shakespeare : [66] first introduced this dataset to FL community.”
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, methodology): “The set of common algorithms includes average and weighted aggregation, FedAvg [18], as described above.”