A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks

By Eden Belouadah, Adrian Daniel Popescu, Ioannis Kanellos. 2020

259 citations recorded by the source corpus.

The ability of artificial agents to increment their capabilities when confronted with new data is an open challenge in artificial intelligence. The main challenge faced in such cases is catastrophic forgetting, i.e., the tendency of neural networks to underfit past data when new ones are ingested. A first group of approaches tackles forgetting by increasing deep model capacity to accommodate new knowledge. A second type of approaches fix the deep model size and introduce a mechanism whose objective is to ensure a good compromise between stability and plasticity of the model. While the first type of algorithms were compared thoroughly, this is not the case for methods which exploit a fixed size model. Here, we focus on the latter, place them in a common conceptual and experimental framework and propose the following contributions: (1) define six desirable properties of incremental learning algorithms and analyze them according to these properties, (2) introduce a unified formalization of the class-incremental learning problem, (3) propose a common evaluation framework which is more thorough than existing ones in terms of number of datasets, size of datasets, size of bounded memory and number of incremental states, (4) investigate the usefulness of herding for past exemplars selection, (5) provide experimental evidence that it is possible to obtain competitive performance without the use of knowledge distillation to tackle catastrophic forgetting and (6) facilitate reproducibility by integrating all tested methods in a common open-source repository. The main experimental finding is that none of the existing algorithms achieves the best results in all evaluated settings. Important differences arise notably if a bounded memory of past classes is allowed or not.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “Past comparative studies [2, 24] found that, while appealing in theory, distillation-based methods un-derperform in EFCIL, particularly for large-scale datasets.”
Long-Tailed Class Incremental Learning (2022, background): “Incremental learning, also known as continual or lifelong learning, is the paradigm of continually learning a sequence of tasks as new data becomes available [5,9,27,29].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Besides, compared with the comparison and summary of the classical continual learning meth-ods [16, 17], in this review, we make a comprehensive summary focusing more on the recently proposed continual learning methods.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “networks (Kemker et al., 2018; Parisi et al., 2019; De Lange et al., 2019; Belouadah et al., 2020), we provide the first comprehensive review that integrates and identifies the gaps between replay in these two fields.”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “, when classes are learned incrementally) [34, 6, 17].”
SS-IL: Separated Softmax for Incremental Learning (2020, methodology): “Also, results in [4] and [6] show that CIL models are sensitive to incremental conditions such as the number of incremental tasks (T ) and exemplar-memory size (|M|).”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “[63] perform a comparison of several classIL methods.”
Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Continual learning [2, 6, 26, 37, 39] aims to equip deep models with the capacity to continuously acquire new knowledge, e.g., learn to recognize new object categories, while handling catastrophic forgetting [18, 27, 28, 32] of the old knowledge.”
Helpful or Harmful: Inter-task Association in Continual Learning (2022, background): “Also, since generated instances may not exactly imitate the distribution of real instances, we may encounter a performance drop [4].”
Deep Class-Incremental Learning: A Survey (2023, background): “Other surveys either focuses on the specific field [25], [26], [27] or lacking the performance evolution among state-of-the-arts [28], [29], [30].”
Class-Incremental Learning: A Survey (2023, background): “Other surveys either focus on the specific field [20], [21] or lack the performance evolution among state-of-the-arts [8], [22], [23].”
Scaling the Number of Tasks in Continual Learning (2022, background): “Thereby, each task is usually seen only once, and agents are evaluated by their average performance on all tasks [Lesort et al., 2020, Lange et al., 2019, Belouadah et al., 2021].”
A Survey on Few-Shot Class-Incremental Learning (2023, unknown): “Belouadah et al. (2021) provide a summary of Class-IL in visual tasks only.”
Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, background): “There are three primary mechanisms used to mitigate forgetting in neural networks [10, 17, 47, 61]: 1) regularizing the plasticity of weights [4, 15, 16, 18, 48, 51, 53, 67, 74, 85], 2) growing the network architecture to accommodate new data [35, 41, 60, 69, 84], and 3) maintaining a subset of prev”
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “This can also be seen in several reviews [4,5,11,29,30,34] reporting few approaches that consider architectures besides CNNs, despite the attempts to investigate CL in RNNs [9, 41].”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “This approach allows to analyze CL while varying the input streams (data perspective ), while in the literature, CL is mostly analyzed from the algorithmic perspective (Lange et al., 2019; Belouadah et al., 2021).”
ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection (2022, background): “The CIL could struggle with the notorious catastrophic forgetting [3], rendering the network losing grasp of the learned knowledge when accepting new tasks, which is also known as the task-recency bias.”
Three types of incremental learning (2022, unknown): “An important challenge in this scenario is learning to discriminate between classes that are not observed together, which has turned out to be very challenging for deep neural networks, especially when storing examples of previously seen classes is not allowed 29,30 .”