SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model

By Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, Yunchao Wei. 2023

207 citations recorded by the source corpus.

The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research. Code has been made available at: https://github.com/GengDavid/SLCA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hang Su
5. Yunchao Wei
6. Jun Zhu
7. Liyuan Wang
8. Xingxing Zhang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Yunchao Wei
4. Da-Wei Zhou
5. Jun Zhu
6. Hang Su
7. Liyuan Wang
8. Xingxing Zhang

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Jun Zhu
8. Hai-Long Sun

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “SLCA [504] observes that slowly fine-tuning the backbone of a pre-trained transformer can achieve outstanding performance in continual learning, and further”
• Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, background): “SLCA [70] extends the Gaussian modeling of previous classes in [80] to rectify classifiers during model updating.”
• Convolutional Prompting meets Language Models for Continual Learning (2024, unknown): “Comparison with SLCA : We also conducted a comparison with a recent SOTA approach Slow Learner with Classifier Alignment (SLCA) [60] that does not use prompts for CL.”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, background): “Another intuitive thought to tackle CIL task is based on parameter isolation [2, 22, 35, 36, 37, 46].”
• Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Therefore, rehearsal-free continual learning meth-ods [9, 23, 25, 35, 43, 51] have attracted much attention.”
• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, background): “Prior studies on continual learning [28, 80, 65, 78, 76] focused on supervised class-incremental learning (CIL), as Figure 1(a), which aims to maintain discriminative features for known classes and expand new classifiers to learn new classes.”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, unknown): “While the research focus of continual learning is on classification [1, 6, 11, 22, 25, 38, 41, 50, 58], continual segmentation started to thrive recently [3, 5, 7, 12, 32, 36, 40, 51, 57, 60, 61, 65].”