Ensemble Distillation for Robust Model Fusion in Federated Learning
By Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi. Neural Information Processing Systems, 2020
1,521 citations recorded by the source corpus.
Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.
Subfields: Artificial Intelligence, Building and Construction, Federated & Distributed Learning
View Paper: https://arxiv.org/abs/2006.07242
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Building and Construction
In Federated & Distributed Learning
- Qiang Yang
- Dacheng Tao
- Bingsheng He
- Qinbin Li
- Murali Annavaram
- Zeyi Wen
- H. Vincent Poor
- Medhi Bennis
Citations in the corpus
- Knowledge Distillation: A Survey (2020, background): “…are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivanetal.2019),interpretabilityofdeepneuralnetworks (Liu et al. 2018b), and federated learning (Bistritz et al. 2020; Lin et al. 2020; Seo et al. 2020; He et al. 2020a).”
- Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, methodology): “In order to solve this problem, a main stream of subsequent effort leverages knowledge transfer through labeled data [38, 74], shared model [48, 72, 92] or group operation [21, 50].”
- Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “We use KD [19] in a different manner from existing and concurrent works [38, 39, 40, 41, 42, 43, 44, 45].”
- Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning (2022, methodology): “We compare FedFTG against FedAvg [30], FedProx [23], SCAFFOLD [18], FedDyn [1], MOON [22], FedGen [43] and FedDF [25].”
- Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, methodology): “Similar to existing works [69, 78], we use Dirichlet distribution with parameter α to allocate examples for this kind of non-IID setting.”
- FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning (2020, background): “We note that, v-Distillation is highly similar to (Lin et al., 2020) except for different hyper-parameters.”
- Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, neutral): “To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30].”
- Federated Adversarial Debiasing for Fair and Transferable Representations (2021, background): “For example, in [10, 14, 21, 26], the consensus on task knowledge is achieved by distillation.”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, methodology): “The work in [129] used ensemble distillation for robust model fusion.”
- Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “To alleviate the reliance on public data, FedDF [135] utilizes unlabeled or generated data for ensemble distillation.”
- Towards Personalized Federated Learning (2021, background): “An α of 100 is equivalent to the IID setting, while a smaller α value means that each client is more likely to hold data from only one class resulting in high data heterogeneity [55, 56, 86].”
- Distributed Distillation for On-Device Learning (2020, background): “In [36] they enable cross-architecture learning by communicating model parameters to a central server, performing FedAvg within model types, then distillation across model types, and finally communicating the updated model parameters back to each device.”
- Data-Free Knowledge Distillation for Heterogeneous Federated Learning (2021, methodology): “, 2019; Chen & Chao, 2021), among which knowledge distillation has emerged as an effective solution (Lin et al., 2020; Li & Wang, 2019).”
- Continual Federated Learning Based on Knowledge Distillation (2022, background): “There are also several works leveraging additional datasets constructed from public dataset [Li and Wang, 2019; Lin et al. , 2020] or original local data [Itahara et al. , 2020] to aid distillation.”