End-to-End Incremental Learning
By Francisco Manuel Castro, Manuel Jesús Marín-Jiménez, Nicolás Guil Mata, Cordelia Schmid, Alahari Karteek. 2018
1,337 citations recorded by the source corpus.
Although deep learning approaches have stood out in recent years due to their state-of-the-art results, they continue to suffer from catastrophic forgetting, a dramatic decrease in overall performance when training with new classes added incrementally. This is due to current neural network architectures requiring the entire dataset, consisting of all the samples from the old as well as the new classes, to update the model—a requirement that becomes easily unsustainable as the number of classes grows. We address this issue with our approach to learn deep neural networks incrementally, using new data and only a small exemplar set corresponding to samples from the old classes. This is based on a loss composed of a distillation measure to retain the knowledge acquired from the old classes, and a cross-entropy loss to learn the new classes. Our incremental training is achieved while keeping the entire framework end-to-end, i.e., learning the data representation and the classifier jointly, unlike recent methods with no such guarantees. We evaluate our method extensively on the CIFAR-100 and ImageNet (ILSVRC 2012) image classification datasets, and show state-of-the-art performance.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Han-Jia Ye
- Bernt Schiele
- De-chuan Zhan
- Dacheng Tao
- Joost van de Weijer
- Bing Liu
- Yoshua Bengio
- Jun Zhu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Yi Yang
- Jitendra Malik
In Radiology, Nuclear Medicine and Imaging
- De-chuan Zhan
- Han-Jia Ye
- Da-Wei Zhou
- Christopher Kanan
- Tyler L. Hayes
- Bernt Schiele
- Ali Ayub
- Alan R. Wagner
In Meta & Continual Learning
- Han-Jia Ye
- De-chuan Zhan
- Da-Wei Zhou
- Joost van de Weijer
- Bing Liu
- Jiangpeng He
- Bernt Schiele
- Gyuhak Kim
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “On the other hand, experience replay can be naturally combined with knowledge distillation (KD), which additionally incorporates the past information from the old model. iCaRL [122] and EEIL [123] are two early works that perform KD on both old and new training samples.”
- S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “EEIL [7] used knowledge distillation to remember old classes and cross-entropy to learn new classes in an end-to-end training.”
- Learnability and Algorithm for Continual Learning (2023, methodology): “The replay methods are: iCaRL (Rebuffi et al., 2017), A-GEM (Chaudhry et al., 2019a), EEIL (Castro et al., 2018), GD (Lee et al., 2019) without external data, DER++ (Buzzega et al., 2020), HAL (Chaudhry et al., 2021), and MORE (Kim et al., 2022a).”
- RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “…task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or class-incremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57].”
- A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “We also consider average incremental accuracy [12] for our analysis, which is the average of the average accuracy across all the tasks.”
- FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “Many recent solutions [4, 13, 33, 44, 46], based on deep nets, use replay from a bounded memory of the past to reduce forgetting.”
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, neutral): “, penalize changes to model predictions) [1,8,23,34,38].”
- A Theoretical Study on Solving Continual Learning (2022, methodology): “Using regularization [12, 13] to minimize changes to model parameters learned from previous tasks is a popular approach [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
- Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Nevertheless, unlike the supervised class incremental learning [7] which typically trains a model to classify on a closed set of categories, VLP models learn an open set of visual concepts from natural language descriptions.”
- Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation (2022, background): “To deal with this problem, the few-shot class-incremental learning (FSCIL) task [17, 29, 4, 35, 44, 48] comes into sight.”
- A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, background): “[2] viewed the first task as a pre-training stage and froze the feature representation after the first task.”
- Few-Shot Continual Active Learning by a Robot (2022, background): “Most existing class-incremental learning (CIL) methods avoid catastrophic forgetting by storing a portion of the training samples from previous classes and retraining the model on a mixture of the stored data and new data [3, 31].”
- Long-Tailed Class Incremental Learning (2022, methodology): “In this section we analyze the performance of four popular CIL methods: LwF [20] with exempalrs, EEIL [7], LUCIR [13], and PODNET [10]).”
- Anti-Retroactive Interference for Lifelong Learning (2022, methodology): “In addition, the distillation loss is implemented by [22,4,36] to reduce forgetting.”
- Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “We compare our method with the SOTA methods [17,1,11,19,4,5,23] on three datasets.”
- Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, background): “Since it’s not capable to properly distinguish between old and new classes, later approaches [2, 19] utilize the memory of old classes for further training by considering distribution relationship between base and new classes.”
- S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.”