Overcoming Forgetting in Federated Learning on Non-IID Data

By N. Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, Itai Zeitak. 2019

283 citations recorded by the source corpus.

We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.

Subfields: Artificial Intelligence, Meta & Continual Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sung Ju Hwang
2. Dacheng Tao
3. Jaehong Yoon
4. Hangyu Zhu
5. Yaochu Jin
6. Eunho Yang
7. Bo Du
8. Qiang Yang

In Meta & Continual Learning

1. Sung Ju Hwang
2. Jaehong Yoon
3. Eunho Yang
4. Dacheng Tao
5. Yaochu Jin
6. Qiang Yang
7. Wonyoung Jeong
8. Giwoong Lee

In Federated & Distributed Learning

1. Qiang Yang
2. Han Yu
3. Li-zhen Cui
4. Hangyu Zhu
5. Jaehong Yoon
6. Dacheng Tao
7. Xiao Wang
8. Bo Du

Citations in the corpus

• Distilled One-Shot Federated Learning (2020, background): “They include modifying the training loss [17], using lifelong learning to prevent forgetting [29], and correcting local updates using control variates [12].”
• Federated Class-Incremental Learning (2022, methodology): “[42] focuses on minimizing the model discrepancies across clients via an improved EWC.”
• Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “3) FedCurv: FCL with FedCurv algorithm (23) which reduces the parameter disparity across clients using Fisher Information Matrix.”
• Continual Federated Learning Based on Knowledge Distillation (2022, background): “Some recent attempts on this topic, such as [ Shoham et al. , 2019; Usmanova et al. , 2021 ] , do not solve the problem adequately since they are designed to address either intra-task forgetting or inter-task forgetting.”
• The future of digital health with federated learning (2020, background): “data at the same time [13], possibly even without forgetting [46].”
• Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “Shortly after, a large body methods [12, 41, 43, 73, 77] research on non-i.”
• Towards Personalized Federated Learning (2021, methodology): “Multi-Task Learning [59]–[62] [59], [61]–[62] [61]–[62] [59, 61] - [59, 61] - Model Interpolation [17], [64]–[65] [17], [63]–[65] [63]–[65] [65] [65] [65] - - Clustering [66]–[70], [72] [66]–[70], [72] [66]–[67], [69]–[70], [72] - - -”
• Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, methodology): “FedCurv [190] and FedCL [229] are adaptations of the Elastic Weight Consolidation (EWC) [103] algorithm to federated learning scenarios.”
• Federated Learning on Non-IID Data: A Survey (2021, methodology): “propose a federated curvature (FedCurv) algorithm based on EWC as a solution to Non-IID issues in FL [119].”