Essentials for Class Incremental Learning

By Sudhanshu Mittal, Silvio Galesso, Thomas Brox. 2021

119 citations recorded by the source corpus.

Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and intertask learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate regularizer. With these lessons learned, class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.

Subfields: Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, methodology): “As also observed in other recent works [28], [29], [42], this issue can be mitigated by revising the way the cross-entropy loss is applied during training.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, background): “Previously, [25] also studied the initial-phase representation for CIL.”
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting (2021, methodology): “The technique is widely used to avoid the explicit calibration of magnitudes [19, 37, 40, 31, 15, 36].”
Class-Incremental Learning via Dual Augmentation (2021, result): “Similar results have also been reported in [61].”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “, [48, 60, 28, 8, 79, 83, 71, 50, 55]) apply knowledge distillation [27] between a network trained on previous tasks and a network being trained on current task to alleviate the performance degradation on previous tasks.”
Efficient Test-Time Model Adaptation without Forgetting (2022, background): “…aims to help the model remember the essential concepts that have been learned previously, alleviating the catastrophic forgetting issue when learning on a new task (Kirkpatrick et al., 2017; Li & Hoiem, 2017; Rolnick et al., 2019; Farajtabar et al., 2020; Niu et al., 2021; Mittal et al., 2021).”
Incremental Prototype Prompt-tuning with Pre-trained Representation for Class Incremental Learning (2022, methodology): “We evaluate the proposed IPP method on various benchmarks, comparing with the reported results of several methods including DER [40], PODNet [6], BiC [38], UCIR [9], CCIL-SD [22], and iCaRL [30].”
Dealing with Cross-Task Class Discrimination in Online Continual Learning (2023, background): “When the new task arrives, we do not need to assume that we have the entire training set for calculating some statistics or an exemplar set like CCIL [33].”