Balancing Stability and Plasticity through Advanced Null Space in Continual Learning

By Yajing Kong, Liu Liu, Zhen Wang, Dacheng Tao. 2022

53 citations recorded by the source corpus.

Continual learning is a learning paradigm that learns tasks sequentially with resources constraints, in which the key challenge is stability-plasticity dilemma, i.e., it is uneasy to simultaneously have the stability to prevent catastrophic forgetting of old tasks and the plasticity to learn new tasks well. In this paper, we propose a new continual learning approach, Advanced Null Space (AdNS), to balance the stability and plasticity without storing any old data of previous tasks. Specifically, to obtain better stability, AdNS makes use of low-rank approximation to obtain a novel null space and projects the gradient onto the null space to prevent the interference on the past tasks. To control the generation of the null space, we introduce a non-uniform constraint strength to further reduce forgetting. Furthermore, we present a simple but effective method, intra-task distillation, to improve the performance of the current task. Finally, we theoretically find that null space plays a key role in plasticity and stability, respectively. Experimental results show that the proposed method can achieve better performance compared to state-of-the-art continual learning approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. René Vidal
6. Paris V. Giampouras
7. Liangzu Peng
8. Tianyu Pang

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. René Vidal
6. Paris V. Giampouras
7. Liangzu Peng
8. Shuicheng Yan

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. René Vidal
6. Paris V. Giampouras
7. Liangzu Peng
8. Simone Scardapane

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Adam-NSCL [66] instead projects candidate parameter updates into the current null space approximated by the uncentered feature covariance of the old tasks, while AdNS [179] considers the shared part of the previous and the current null spaces.”
• The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “(2021b) are in principle equivalent, and their difference in performance (as reported by Kong et al. (2022)) relies on implementation details. Furthermore, their approaches are dual to each other and are the two sides of the same coin: Wang et al. (2021b) (approximately) maintains a basis for ∩t−1 i”