Continual learning: A comparative study on how to defy forgetting in classification tasks

By Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, G. Slabaugh, Tinne Tuytelaars. 2019

260 citations recorded by the source corpus.

Artificial neural networks thrive in solving the classification problem for a particular rigid task, where the network resembles a static entity of knowledge, acquired through generalized learning behaviour from a distinct training phase. However, endeavours to extend this knowledge without targeting the original task usually result in a catastrophic forgetting of this task. Continual learning shifts this paradigm towards a network that can continually accumulate knowledge over different tasks without the need for retraining from scratch, with methods in particular aiming to alleviate forgetting. We focus on task-incremental classification, where tasks arrive in a batch-like fashion, and are delineated by clear boundaries. Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 10 state-of-the-art continual learning methods and 4 baselines. We empirically scrutinize which method performs best, both on balanced Tiny Imagenet and a large-scale unbalanced iNaturalist datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time and storage.

Subfields: Meta & Continual Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

In Statistical Learning Theory

Citations in the corpus

RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “Existing methods tackling this can be divided into three categories: memory-based, regularization-based, and network-architecture-based [11, 35].”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “This is an important limitation because data often occur in sequences [17] and their storage is costly.”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, background): “Previous works can be roughly grouped in two categories [18], i.”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “Despite progress in continual learning (Lange et al., 2019; Parisi et al., 2019; Biesialska et al., 2020; Khetarpal et al., 2020a; Hadsell et al., 2020), there is – to the best of our knowledge – no work that addresses continual reinforcement learning (continual RL) of the dialogue policy, even…”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, methodology): “We now evaluate CAT following the standard continual learning evaluation method in (Lange et al., 2019).”
Self-Supervised Models are Continual Learners (2021, methodology): “We choose baselines from three categories [17]: priorfocused regularization (EWC [31]), data-focused regularization (POD [18], Less-Forget [30]), and rehearsal-based replay (ER [44], DER [4]) methods.”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, methodology): “We follow the continual learning (CL) evaluation method given in [29].”
Continual Learning via Local Module Composition (2021, background): “The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, methodology): “Aiming to mitigate the Catastrophic Forgetting problem, continual learning has been widely discussed on image clas-siﬁcation [23, 7], which can be divided into replay methods, parameter isolation methods and regularization-based meth-ods [8].”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “We follow the standard CL evaluation method in (Lange et al., 2019).”
Continual Learning for Named Entity Recognition (2021, methodology): “The goal of CL is to gradually extend the knowledge acquired in a model (Lange et al. 2019), without incurring any Catastrophic Forgetting (CF) (Goodfellow et al.”
Rectification-based Knowledge Retention for Continual Learning (2021, methodology): “To avoid catastrophic forgetting, deep learning models should perform lifelong/continual/incremental learning [5, 18].”
Self-Supervised Training Enhances Online Continual Learning (2021, neutral): “, when classes are learned incrementally) [34, 6, 17].”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, methodology): “Approaches As previously discussed, current CL methods can be classified into three major categories: regularization, parameter-isolation, and replay methods [43, 14].”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, methodology): “In order to set the hyper-parameters of each method, we follow the same continual learning protocol of [10, 4], i.”