Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning

By Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, Ling-Yu Duan. Computer Vision and Pattern Recognition, 2022

428 citations recorded by the source corpus.

Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. Data heterogeneity is one of the main challenges in FL, which results in slow convergence and degraded performance. Most existing approaches only tackle the heterogeneity challenge by restricting the local model update in client, ignoring the performance drop caused by direct global model aggregation. Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation. Concretely, FedFTG explores the input space of local models through a generator, and uses it to transfer the knowledge from local models to the global model. Besides, we propose a hard sample mining scheme to achieve effective knowledge distillation throughout the training. In addition, we develop customized label sampling and class-level ensemble to derive maximum utilization of knowledge, which implicitly mitigates the distribution discrepancy across clients. Extensive experiments show that our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.

Subfields: Artificial Intelligence, Building and Construction, Computer Science Applications

View Paper: https://www.semanticscholar.org/paper/d3823114651475280571cc7af95809dd739d529f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Du
2. Mang Ye
3. Pong Chi Yuen
4. Dacheng Tao
5. Xiuwen Fang
6. Jing Zhang
7. Charu C. Aggarwal
8. Xinlei Chen

In Building and Construction

1. Dacheng Tao
2. Bo Du
3. Mang Ye

In Computer Science Applications

1. Dacheng Tao
2. Bo Du
3. Mang Ye

Citations in the corpus

• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “FedFTG [245] trains a conditional generator to fit the input space of a local model, which is then used to generate pseudo data.”