Communication-efficient federated learning via knowledge distillation

By Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, Xing Xie. Nature Communications, 2021

593 citations recorded by the source corpus.

Federated learning is a privacy-preserving machine learning technique to train intelligent models from decentralized data, which enables exploiting private data by communicating local model updates in each iteration of model learning rather than the raw data. However, model updates can be extremely large if they contain numerous parameters, and many rounds of communication are needed for model training. The huge communication cost in federated learning leads to heavy overheads on clients and high environmental burdens. Here, we present a federated learning method named FedKD that is both communication-efficient and effective, based on adaptive mutual knowledge distillation and dynamic gradient compression techniques. FedKD is validated on three different scenarios that need privacy protection, showing that it maximally can reduce 94.89% of communication cost and achieve competitive results with centralized model learning. FedKD provides a potential to efficiently deploy privacy-preserving intelligent systems in many scenarios, such as intelligent healthcare and personalization. This work presents a communication-efficient federated learning method that saves a major fraction of communication cost. It reveals the advantage of reciprocal learning in machine knowledge transfer and the evolutional low-rank properties of deep model updates.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/56708004ada46632aa25d1d7a3f2d5b92b54166f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lichao Sun
2. Lingjuan Lyu
3. Philip S. Yu
4. Qiang Yang
5. Dacheng Tao
6. Bo Du
7. Han Yu
8. Mang Ye

In Information Systems

1. Lingjuan Lyu
2. Philip S. Yu
3. Jun Zhao
4. Dacheng Tao
5. Bo Du
6. Qiang Yang
7. Han Yu
8. Mang Ye

Citations in the corpus

• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “Federated learning has drawn increasing attention in both academia and industry owing to its potential utility in large-scale applications [48, 85, 185], which has been widely explored in the fields of healthcare [39, 178, 214], medical analysis [54, 97], and data security [205], etc.”
• Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges (2022, background): “• Use of meta-learning, multitasking, and federated distillation in federation tasks [91], [111]–[114]”
• Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “In recent years, FL has benefited a wide range of applications such as next word prediction [6], [7], visual object detection for safety [8], entity resolution [9], recommendation [10], [11], [12], industrial IoT [13], and graph-based analysis [14], [15], [16], etc.”