FedMD: Heterogenous Federated Learning via Model Distillation
By Daliang Li, Junpu Wang. 2019
1,187 citations recorded by the source corpus.
Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.
Subfields: Artificial Intelligence, Computer Science Applications, Sociology and Political Science, Efficiency & Compression, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Science Applications
- Shiqing Liu
- Jinjin Xu
- Yaochu Jin
- Zachary B. Charles
- Hong-You Chen
- Wei-Lun Chao
- Hangyu Zhu
- Jiayu Zhou
In Sociology and Political Science
In Efficiency & Compression
- Murali Annavaram
- S. Avestimehr
- Chaoyang He
- Dacheng Tao
- Li-zhen Cui
- Hangyu Zhu
- Hong-You Chen
- Zachary B. Charles
In Federated & Distributed Learning
Citations in the corpus
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “In terms of training procedures, the training procedures in federated GAN [59, 60] and transfer learning-based FL [61, 62, 63, 64, 65] are significantly different from the vanilla FedAvg algorithm [66].”
- Distilled One-Shot Federated Learning (2020, background): “While effective, breaking from this pattern offers many benefits, such as low communication [9] or private model architectures [15].”
- Federated Model Distillation with Noise-Free Differential Privacy (2020, methodology): “More recently, FedMD [Li and Wang, 2019] and Cronus [Chang et al.”
- Adaptive Personalized Federated Learning (2020, background): “This is even more crucial in fairness-critical systems such as medical diagnosis (Li and Wang, 2019), where poor performance on local clients could result in damaging consequences.”
- Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, methodology): “solution to this problem as highlighted in [75].”
- Towards Personalized Federated Learning (2021, methodology): “Learning in the presence of concept drift often involve three steps: (i) drift detection (whether drift has occurred); (ii) drift understanding (when, how, and where the drift occurs); and (iii) drift adaptation (response to drift) [99].”
- Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “FL with Heterogeneous Architectures: The most recent efforts extended FL to collaboratively train models with heterogeneous architectures [15], [16].”
- On Large-Cohort Training for Federated Learning (2021, background): “…McMahan et al., 2017; Basu et al., 2019; Laguel et al., 2021), data and systems heterogeneity (Li et al., 2020a, 2019; Karimireddy et al., 2020b; Hsu et al., 2019; Karimireddy et al., 2020a; Li et al., 2020b; Wang et al., 2020; Li and Wang, 2019), and fairness (Li et al., 2020c; Hu et al., 2020).”
- Membership Inference Attacks on Machine Learning: A Survey (2021, background): “On the contrary, heterogeneous federated learning schemes such as FedMD [109] and HeteroFL [39] have recently been proposed to handle the system heterogeneity [111] without requiring local models to share the same architecture.”
- Survey of Personalization Techniques for Federated Learning (2020, background): “Li et al. [31] propose FedMD, a federated learning framework based on knowledge distillation and transfer learning that allows clients to independently design their own networks us-ing their local private datasets and a global public dataset.”
- Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “The basic ideas of Cronus [21] and FedMD are similar as they both operate on public data via knowledge distillation.”
- Federated Adversarial Debiasing for Fair and Transferable Representations (2021, background): “For example, in [10, 14, 21, 26], the consensus on task knowledge is achieved by distillation.”
- FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, methodology): “In MNIST and FEMNIST, the number of output channels in the convolutional layers is set to either 18, 20 or 22, while in CIFAR10, the stride of convolutional layers is set differently across different clients.”
- Federated Mutual Learning (2020, background): “Li and Wang [14] proposes a decentralized framework based on knowledge distillation, which enables federated learning for independently designed models.”