A Continual Learning Survey: Defying Forgetting in Classification Tasks
By Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, G. Slabaugh, Tinne Tuytelaars. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
2,435 citations recorded by the source corpus.
Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern: (1) a taxonomy and extensive overview of the state-of-the-art; (2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner; (3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods; and (4) baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning
View Paper: https://doi.org/10.1109/TPAMI.2021.3057446
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Bernt Schiele
- Yi Yang
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
In Meta & Continual Learning
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “General Continual Learning (GCL) [16], [46], [85] is a broad concept that describes potential challenges in terms of learning paradigms, where the model observes incremental data in an online fashion without explicit task boundaries.”
- Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, methodology): “The CUB-200-2011 dataset [62] includes 200-class bird images of size 224 × 224, and the Tiny-ImageNet dataset [19] includes 200-class natural images of size 64 × 64, both split randomly into 10 tasks.”
- The Ideal Continual Learner: An Agent That Never Forgets (2023, neutral): “…Ramesh & Chaudhari, 2022; Douillard et al., 2022), or combinations thereof (Yan et al., 2021; Xie et al., 2022; Wang et al., 2022b; Frascaroli et al., 2023); for surveys, see, e.g., Parisi et al. (2019); De Lange et al. (2022); Qu et al. (2021); Shaheen et al. (2022); van de Ven et al. (2022).”
- Towards Label-Efficient Incremental Learning: A Survey (2023, background): “While there have been few studies that surveys conventional incremental learners [Masana et al. , 2020; De Lange et al. , 2021; Wang et al. , 2023; Zhou et al. , 2023], their focus is still on the old label-hungry paradigm in incremental learning.”
- Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Continual Learning Continual learning [54,13,43,48] aims to integrate novel knowledge in a sequential fashion where old data are usually unavailable.”
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “W.r.t. the latter, TIL has been criticized as a less challenging and realistic benchmark [26, 74, 4]; we therefore conduct our main experiments on state-of-the-art CL models in the CIL setting 1 .”
- Class Gradient Projection For Continual Learning (2022, background): “Such a phenomenon is known as catastrophic forgetting , which is the research hotspot in Continual Learning (CL) [11, 28, 32].”
- Long-Tailed Class Incremental Learning (2022, background): “Incremental learning, also known as continual or lifelong learning, is the paradigm of continually learning a sequence of tasks as new data becomes available [5,9,27,29].”
- Disentangling Transfer in Continual Reinforcement Learning (2022, background): “[6, 7]; see also CL survey papers [8, 17, 32].”
- Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, background): “Experience replay (ER), also known as rehearsal [Chaudhry et al., 2019, Delange et al., 2021], is a key idea in OCL.”
- DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Existing works on continual learning can be largely categorized into three groups: model regularization, data rehearsal, and parameter isolation [7].”
- One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, neutral): “Another related setting is continual learning where batches of data from different tasks are sequentially learned [6].”
- Online Continual Learning with Contrastive Vision Transformer (2022, background): “Continual learning (also known as lifelong learning or incremental learning) [62,63,79,75,80,95,18] aims to solve this problem by maintaining and accumulating the acquired knowledge over time from a stream of non-stationary data.”
- Incremental Task Learning with Incremental Rank Updates (2022, background): “Incremental task learning (ITL) [10,34] aims to train a single model on a sequence of different tasks and perform well on all the trained tasks once the training is finished.”
- Class-incremental Novel Class Discovery (2022, background): “Although the model gets better at classifying the novel classes, it’s performance gradually deteriorates on the base classes due to forgetting [8].”
- Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”