Federated Learning of Deep Networks using Model Averaging

By H. Brendan McMahan, Eider Moore, Daniel Ramage, Blaise Agüera y Arcas. 2016

1,281 citations recorded by the source corpus.

Subfields: Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Federated & Distributed Learning

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, methodology): “We employ the Federated Averaging algorithm [13], which significantly decreases the number of rounds of communication required to train a good model.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “One practical setting is user privacy (Shokri and Shmatikov, 2015; McMahan et al., 2016).”
Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, neutral): “, averaging on these parameters [6]–[8].”
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Google has recently introduced Federated Learning [48, 49] to enable mobile devices to collaboratively learn a shared prediction model while keeping all the train-”
Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, background): “In (McMahan et al., 2016), data samples are distributed by different participants thus it can be regarded as sample-partitioned collaborative learning.”
Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, methodology): “• To aggregate the information learned from each of the clients into a global model using classical federated aggregation operators such as: FedAvg, weighted FedAvg [24] and an aggregation operator for the adaptation of the k-means algorithm to a federated setting [25].”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, background): “Local SGD is a simpliﬁed version of the federated averaging algorithm (McMahan et al., 2016; 2017) and has recently attracted the attention of the theoretical community in the seek of the best convergence rates (Stich, 2019b; Wang & Joshi, 2018; Yu et al., 2019; Basu et al., 2019; Patel &…”
A Communication Efficient Vertical Federated Learning Framework (2019, background): “In [1], it is demonstrated experimentally that multiple local updates can be performed in HFL with federated averaging (FedAvg), reducing the number of communication round effectively.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, background): “For example, federated learning (Konevcnỳ et al., 2016; McMahan et al., 2016) adopts local SGD with heterogeneous numbers of local steps and subsampling workers to train models on edge devices.”
Real-World Image Datasets for Federated Learning (2019, methodology): “We adapt the original FederatedAveraging ( FedAvg ) algorithm [McMahan et al. , 2016] to framework, shown in Algorithm 1.”
Detailed comparison of communication efficiency of split learning and federated learning (2019, background): “Federated learning (McMahan et al., 2016; Koneˇcn`y et al., 2016) and Split learning (Gupta & Raskar, 2018; Vepakomma et al., 2018, 2019) are two methods which allow to train a model collectively from various distributed data sources without sharing raw data.”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, background): “This especially covers learning over decentralized data, as extensively studied in the federated learning literature for the centralized algorithms (McMahan et al., 2016).”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some references are Konecný et al. (2015, 2016); McMahan et al. (2016).”
Federated Machine Learning (2019, background): “In [41], the authors proposed building a secure client–server structure in which the federatedlearning system partitions data by users and allow models built at client devices to collaborate at the server site to build a global federated model.”
Federated Learning for Keyword Spotting (2018, methodology): “In this work, we investigate the use of federated learning (FL) [3] in the context of an embedded wake word detector.”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Distributed stochastic gradient descent (SGD) is a basic building block of modern machine learning [25, 11, 9, 28, 1, 27, 5].”
Randomized Distributed Mean Estimation: Accuracy vs. Communication (2016, background): “It has recently been observed that when communication cost is high, which is typically the case for commodity clusters, and even more so in a federated optimization framework, it is can be very useful to sacrifice on estimation accuracy in favor of reduced communication [7, 4].”