Federated Continual Learning with Weighted Inter-client Transfer

By Jaehong Yoon, Wonyoung Jeong, Giwoong Lee, Eunho Yang, Sung Ju Hwang. 2020

322 citations recorded by the source corpus.

There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in real-world scenarios. Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from a private local data stream. This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge. To resolve these issues, we propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT), which decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-specific parameters.FedWeITminimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning. We validate ourFedWeITagainst existing federated learning and continual learning methods under varying degrees of task similarity across clients, and our model significantly outperforms them with a large reduction in the communication cost.

Subfields: Artificial Intelligence, Meta & Continual Learning, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

In Federated & Distributed Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “…data collected by clients is dynamic and variable, federated learning needs to overcome catastrophic forgetting and facilitate knowledge transfer across clients, which can be achieved by continual learning strategies such as model decomposition [304] and knowledge distillation [305], [306].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[77] proposed a new problem setup called federated continual learning, which allows several clients to carry out continual learning each by itself through an independent stream of data instances that is inaccessible to other clients.”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “Coarse-CIFAR100 is a variant of CIFAR100 where each super-class is considered a diﬀerent task (Yoon et al., 2019; 2021; Shanahan et al., 2021).”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, neutral): “In light of this need, continual learning (CL) [40], or lifelong learning, tackles a learning scenario where a model continuously learns over a sequence of tasks [18, 25] within a broad research area, such as classification [15, 9], image generation [47], language learning [24, 4], clinical applicat”
CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability (2021, methodology): “For example, without delving into mathematical details, it suffices to know that FedWeIT prioritizes a trade-off with respect to communication, parameter and memory efficiency in a federated setting.”
Continual Federated Learning Based on Knowledge Distillation (2022, background): “[Yoon et al. , 2021] proposed Federated Continual Learning and focused on multiple continual learning agents that use each other’s indirect experience to enhance the continual learning performance of their local models, rather than to jointly train a better global model.”
TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, methodology): “FedWeIT solves these challenges through decomposing the parameters into three parts, i.e., global parameters, local based parameters, and task-adaptive parameters [58].”
Decentralized Federated Learning: A Survey and Perspective (2023, background): “A number of concepts and algorithms for federated continual learning are mentioned in the recent literature [157], [158], [159], [160], which consider the process of dynamic data collection in the real world while addressing the issues of non-IID data, concept drift, and catastrophic forgetting.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “In a distinct line of work, Yoon et al. (2021) developed a knowledge-sharing mechanism for lifelong federated learning, selectively transferring knowledge across clients.”