Federated Learning with Matched Averaging
By Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, Yasaman Khazaeni. 2020
1,404 citations recorded by the source corpus.
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, while improving the communication efficiency.
Subfields: Artificial Intelligence, Building and Construction, Optimization & Training, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Building and Construction
In Optimization & Training
- Qiang Yang
- Martin Jaggi
- Aurélien Bellet
- Peter Richtárik
- Ji Liu
- Chaoyang He
- Sebastian U. Stich
- Wei-Lun Chao
In Federated & Distributed Learning
- Qiang Yang
- Yang Liu
- Chaoyang He
- Murali Annavaram
- Bingsheng He
- Qinbin Li
- Amir Salman Avestimehr
- Tianjian Chen
Citations in the corpus
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, neutral): “, 2019; 2020), and 4) FEDMA3 (Wang et al., 2020) (for better model fusion).”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “Following [138] and [2], we use latent Dirichlet allocation (LDA) to partition the dataset according to the number of workers involved in training in each round.”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, neutral): “1) , as done in [55].”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Alternatives to simple weighted averaging have also been proposed: PFNM [47] and FedMA [48] proposes using neural matching, and [49] uses optimal transport to achieve model aggregation.”
- The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “Several recent works show significant accuracy loss for FederatedAveraging over non-IID data, and some propose algorithms to improve FederatedAveraging over non-IID data (Zhao et al., 2018; Li et al., 2019; Shoham et al., 2019; Karimireddy et al., 2019; Liang et al., 2019; Li et al., 2020a; Wang et ”
- SecureBoost: A Lossless Federated Learning Framework (2019, background): “Following the same idea, several attempts have been made to reinvent different machine learning models to the federated setting, including decision tree [20], [21], linear/logistic regression [22], [23], [24] and neural network [25], [26].”
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, methodology): “FL approaches (FedAvg [6], FedProx [10], and FedMA [8]) can reduce communication frequency by local SDG and model averaging [11], Preprint.”
- Model Fusion via Optimal Transport (2019, background): “In a concurrent work 2 , Wang et al. [19] extend [18] to more realistic networks including CNNs, also with a specific focus on federated learning.”
- Federated Class-Incremental Learning (2022, background): “Federated Learning (FL) is a decentralized learning framework that can train a global model by aggregating lo-cal model parameters [24, 44, 51, 53].”
- Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “Recent works (13; 24) introduce well-designed aggregation policies by leveraging Bayesian non-parametric methods.”
- Personalized Cross-Silo Federated Learning on Non-IID Data (2020, background): “Global federated learning (Ji et al. 2019; McMahan et al. 2016; Wang et al. 2020; Yurochkin et al. 2019) trains a single global model to minimize an empirical risk function over the union of the data across all clients.”
- Federated Multi-Task Learning under a Mixture of Distributions (2021, methodology): “[59] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.”
- Federated Learning for Open Banking (2021, background): “A proper neuron matching mechanism in the context of federated learning can improve the performance [33].”
- Dataless Knowledge Fusion by Merging Weights of Language Models (2022, neutral): “Further, improved merging algorithms aim to match permutations between the weights of different models (Singh & Jaggi, 2020; Nguyen et al., 2021; Ainsworth et al., 2022; Wang et al., 2020a), which is a complementary line of effort to our work.”
- FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, methodology): “FedMA (Wang et al., 2020) proposed an aggregation strategy for non-iid data partition that shares global model in a layer-wise manner.”
- Federated Learning on Non-IID Data Silos: An Experimental Study (2021, methodology): “, a large number of local epochs) [31, 39, 61, 62].”
- Server Averaging for Federated Learning (2021, background): “Another approach is to proposes different averaging schemes to either save the communication cost or to improve the performance [14].”
- Federated Learning on Non-IID Data: A Survey (2021, methodology): “Since the local data distributions are different from the global data distribution, the averaged local model parameters may also be far away from the global model parameters [166], particularly when the number of epochs for local updates is large [62, 110, 130].”