Striking a Balance between Stability and Plasticity for Class-Incremental Learning

By Guile Wu, Shaogang Gong, Pan Li. 2021

65 citations recorded by the source corpus.

Class-incremental learning (CIL) aims at continuously updating a trained model with new classes (plasticity) without forgetting previously learned old ones (stability). Contemporary studies resort to storing representative exemplars for rehearsal or preventing consolidated model parameters from drifting, but the former requires an additional space for storing exemplars at every incremental phase while the latter usually shows poor model generalization. In this paper, we focus on resolving the stability-plasticity dilemma in class-incremental learning where no exemplars from old classes are stored. To make a trade-off between learning new information and maintaining old knowledge, we reformulate a simple yet effective baseline method based on a cosine classifier framework and reciprocal adaptive weights. With the reformulated baseline, we present two new approaches to CIL by learning class-independent knowledge and multi-perspective knowledge, respectively. The former exploits class-independent knowledge to bridge learning new and old classes, while the latter learns knowledge from different perspectives to facilitate CIL. Extensive experiments on several widely used CIL benchmark datasets show the superiority of our approaches over the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Hang Su
3. Liyuan Wang
4. Jiashi Feng
5. Xingxing Zhang
6. Rahaf Aljundi
7. Philip H. S. Torr
8. Xialei Liu

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Jun Zhu
3. Hang Su
4. Luc Van Gool
5. Dacheng Tao
6. Philip H. S. Torr
7. Liyuan Wang
8. Bernt Schiele

In Meta & Continual Learning

1. Liyuan Wang
2. Xingxing Zhang
3. Hang Su
4. Jun Zhu
5. Rahaf Aljundi
6. Xialei Liu
7. Philip H. S. Torr
8. Wei Wei

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Other methods exploit the class-wise statistics of feature representations to obtain a balanced classifier, such as by imposing the representations to be transferable and invariant (SPB [453], IL2A [509], SSRE [512], FeTrIL [333]), or compensating explicitly the representation shifts (SDC [484], RER”
• First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “Some class incremental learning methods leverage large initial training session or start from a pre-trained network, e.g., [35, 9, 36] and show improved over-all continual learning performance without questioning the role of such pre-training steps and whether the learned representations have in…”
• Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Some work propose to learn better representations by combining selfsupervised learning for feature extractor [9, 87] and solving class imbalance [39, 46, 54, 99, 102].”
• Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “To mitigate this dilemma, some methods have been proposed to maintain a good balance be-tween old and new classes [4,19,32,33].”
• CGLB: Benchmark Tasks for Continual Graph Learning (2022, background): “In the past, continual learning has been explored in many areas including computer vision [19, 51], reinforcement learning [38], etc. Essentially, it targets at resolving the catastrophic forgetting problem, i.e. the drastic decrease of a model’s performance on previous tasks after being trained on…”
• Bring Evanescent Representations to Life in Lifelong Class Incremental Learning (2022, methodology): “An exemplar-free approach using self-supervised learning was devised in [35], yet with focus on embedding networks.”
• Balancing Between Forgetting and Acquisition in Incremental Subpopulation Learning (2022, methodology): “NEB methods mostly aim to design better forgetting regularization constraints on parameters [14,40,19,2,39] and model outputs [8,41,35,15].”