Class-Incremental Learning with Strong Pre-trained Models

By Tz-Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, Stefano Soatto. 2022

94 citations recorded by the source corpus.

Class-incremental learning (CIL) has been widely stud-ied under the setting of starting from a small number of classes (base classes). Instead, we explore an understud-ied real-world setting of CIL that starts with a strong model pre-trained on a large number of base classes. We hypoth-esize that a strong base model can provide a good repre-sentation for novel classes and incremental learning can be done with small adaptations. We propose a 2-stage training scheme, i) feature augmentation - cloning part of the backbone and fine-tuning it on the novel data, and ii) fusion - combining the base and novel classifiers into a unified classifier. Experiments show that the proposed method sig-nificantly outperforms state-of-the-art CIL methods on the large-scale ImageNet dataset (e.g. + 10% overall accuracy than the best). We also propose and analyze understudied practical CIL scenarios, such as base-novel overlap with distribution shift. Our proposed method is robust and gen-eralizes to all analyzed CIL settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “In addition to an earlier work on obtaining sparse representations through meta-training [185], recent works have attempted to incorporate the advantages of selfsupervised learning (SSL) [125], [281], [335] and large-scale pre-training [295], [380], [456] to improve the representations in initializa”
Learnability and Algorithm for Continual Learning (2023, methodology): “We could not make the recent system in (Wu et al., 2022) using a pre-trained model as no code is released.”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “Finally, we introduce an additional baseline adapter inspired by [37], called FSA-LL (Last Layer).”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “Also, learning a large number of base classes in the initial training phase allows class-incremental learning with small adaptations [48].”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “, 2021a) or get multi-head predictions (Wu et al., 2022), which requires an extra tuning process or routing design.”
Computationally Budgeted Continual Learning: What Does Matter? (2023, background): “In addition, we examine the role of pretraining which has recently become a widely popular direction for exploration in continual learning [48].”