Continual Federated Learning Based on Knowledge Distillation

By Yuhang Ma, Zhongle Xie, Jue Wang, Ke Chen, Lidan Shou. 2022

132 citations recorded by the source corpus.

Federated learning (FL) is a promising approach for learning a shared global model on decentralized data owned by multiple clients without exposing their privacy. In real-world scenarios, data accumulated at the client-side varies in distribution over time. As a consequence, the global model tends to forget the knowledge obtained from previous tasks while learning new tasks, showing signs of "catastrophic forgetting". Previous studies in centralized learning use techniques such as data replay and parameter regularization to mitigate catastrophic forgetting. Unfortunately, these techniques cannot adequately solve the non-trivial problem in FL. We propose Continual Federated Learning with Distillation (CFeD) to address catastrophic forgetting under FL. CFeD performs knowledge distillation on both the clients and the server, with each party independently having an unlabeled surrogate dataset, to mitigate forgetting. Moreover, CFeD assigns different learning objectives, namely learning the new task and reviewing old tasks, to different clients, aiming to improve the learning ability of the model. The results show that our method performs well in mitigating catastrophic forgetting and achieves a good trade-off between the two objectives.

Subfields: Meta & Continual Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Weiming Zhuang
6. Chen Chen
7. Jie M. Zhang
8. LingJuan Lv

In Federated & Distributed Learning

1. Weiming Zhuang
2. Chen Chen
3. Jie M. Zhang
4. LingJuan Lv

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “…data collected by clients is dynamic and variable, federated learning needs to overcome catastrophic forgetting and facilitate knowledge transfer across clients, which can be achieved by continual learning strategies such as model decomposition [304] and knowledge distillation [305], [306].”
• TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, background): “For example, Ma et al. [36] utilize an unlabeled surrogate dataset to address the catastrophic forgetting problem, which may be difficult to obtain in some data sensitive scenarios.”