Class-Incremental Learning: Survey and Performance Evaluation on Image Classification
By Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, Joost van de Weijer. 2020
891 citations recorded by the source corpus.
For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored – also important when privacy limitations are imposed; and learning that more closely resembles human learning. The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one. Incremental learning of deep neural networks has seen explosive growth in recent years. Initial work focused on task-incremental learning, where a task-ID is provided at inference time. Recently, we have seen a shift towards class-incremental learning where the learner must discriminate at inference time between all classes seen in previous tasks without recourse to a task-ID. In this paper, we provide a complete survey of existing class-incremental learning methods for image classification, and in particular, we perform an extensive experimental evaluation on thirteen class-incremental methods. We consider several new experimental scenarios, including a comparison of class-incremental methods on multiple large-scale image classification datasets, an investigation into small and large domain shifts, and a comparison of various network architectures.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Joost van de Weijer
- Han-Jia Ye
- De-chuan Zhan
- Tinne Tuytelaars
- Da-Wei Zhou
- Bing Liu
- Christopher Kanan
- Bartłomiej Twardowski
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Trevor Darrell
- Yi Yang
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
In Radiology, Nuclear Medicine and Imaging
- Joost van de Weijer
- Adrian Daniel Popescu
- Eden Belouadah
- Da-Wei Zhou
- De-chuan Zhan
- Han-Jia Ye
- Gyuhak Kim
- Marc Masana
In Meta & Continual Learning
- Joost van de Weijer
- Bartłomiej Twardowski
- Zixuan Ke
- Tinne Tuytelaars
- Bing Liu
- Jiangpeng He
- Feng Zhu
- Gyuhak Kim
Citations in the corpus
- Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Motivated by these challenges, researchers put increasing attention on incremental (or continual) learning [Masana et al. , 2020 ] .”
- FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “Past comparative studies [2, 24] found that, while appealing in theory, distillation-based methods un-derperform in EFCIL, particularly for large-scale datasets.”
- Long-Tailed Class Incremental Learning (2022, methodology): “1: An illustration of our proposed long-tailed CIL (LT-CIL) scenarios compared to conventional CIL [27] with balanced distribution and few-shot CIL [37].”
- Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “A wide variety of such replay methods have been developed recently, and have attained promising results in the CIL setting.”
- Class-incremental Novel Class Discovery (2022, methodology): “We refer the readers to the survey in [30] for an exhaustive list of class-IL meth-ods.”
- DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “However, the task-incremental setting is usually considered over-simplified [3, 33, 36], since task identity is not known at test time in the real world.”
- FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “In [39], instances are synthesized by a generative model [16] for rehearsal [30] test various exemplar selection strategies, showing that different ways of exemplar selection have a significant impact on performance and herding surpass other strategies in most settings.”
- Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “machine learning field [25], [26], [27], [28], [29], which can be”
- Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “Two classic setups of Incremental Learning are Class In-cremental Learning (CIL) [1,12,13,19,20,22,26] and Task Incremental Learning (TIL) [2, 16, 21, 24, 28, 30, 31].”
- Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “Development of models that support few-shot class-incremental learning (FSCIL), in which classifiers’ label sets can be easily extended with small numbers of new examples and no retraining, is a key challenge for machine learning systems deployed in the real world (Masana et al., 2020).”
- Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Compared to existing continual learning overviews [13–15] focusing on image classification, we find that continual learning has been recently applied in various other computer vision subareas.”
- Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Recent works have revealed that the Softmax classifier and its associated fullyconnected (FC) layer are seriously affected by the class imbalance, which leads to task-recency bias — the tendency of a model to be biased towards classes from the most recent task [39, 37, 24, 57].”
- IIRC: Incremental Implicitly-Refined Classification (2020, methodology): “The replay and regularization based approaches can be used with both task and class incremental setups, however, replay based approaches usually perform better in the class incremental setup [31].”
- MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “R ECENT years have witnessed a great development of class-incremental learning [1], [2], [3], [4], [5], [6], [7], [8], which aims at enabling a learner to acquire new knowledge from new data while preserving the learned knowledge from previous data.”
- Avalanche: an End-to-End Library for Continual Learning (2021, background): “…last few years, machine learning has witnessed a prolific, variegated, and original research production on the topic: from Computer Vision [9, 30, 36] Figure 1: Operational representation of Avalanche with its main modules (top), the main object instances (middle) and the generated stream of…”
- Not Just Selection, but Exploration: Online Class-Incremental Continual Learning via Dual View Consistency (2022, background): “In order to achieve a higher level intelligent system, Continual Learning [17,35,38] studies the problem of learning from a never-ending data stream.”