A Continual Learning Survey: Defying Forgetting in Classification Tasks

By Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, G. Slabaugh, Tinne Tuytelaars. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

2,435 citations recorded by the source corpus.

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern: (1) a taxonomy and extensive overview of the state-of-the-art; (2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner; (3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods; and (4) baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

View Paper: https://doi.org/10.1109/TPAMI.2021.3057446

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dacheng Tao
  2. Han-Jia Ye
  3. Jun Zhu
  4. De-chuan Zhan
  5. Mohit Bansal
  6. Yoshua Bengio
  7. Bernt Schiele
  8. Sergey Levine

In Computer Vision and Pattern Recognition

  1. Dacheng Tao
  2. Luc Van Gool
  3. Bernt Schiele
  4. Yi Yang
  5. Trevor Darrell
  6. Xiaogang Wang
  7. Ming-Hsuan Yang
  8. Andrew Zisserman

In Meta & Continual Learning

  1. Liyuan Wang
  2. Joost van de Weijer
  3. Han-Jia Ye
  4. De-chuan Zhan
  5. Xingxing Zhang
  6. Jun Zhu
  7. Hang Su
  8. Da-Wei Zhou

Citations in the corpus