NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks

By Mustafa Burak Gurbuz, C. Dovrolis. 2022

57 citations recorded by the source corpus.

The goal of continual learning (CL) is to learn different tasks over time. The main desiderata associated with CL are to maintain performance on older tasks, leverage the latter to improve learning of future tasks, and to introduce minimal overhead in the training process (for instance, to not require a growing model or retraining). We propose the Neuro-Inspired Stability-Plasticity Adaptation (NISPA) architecture that addresses these desiderata through a sparse neural network with fixed density. NISPA forms stable paths to preserve learned knowledge from older tasks. Also, NISPA uses connection rewiring to create new plastic paths that reuse existing knowledge on novel tasks. Our extensive evaluation on EMNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets shows that NISPA significantly outperforms representative state-of-the-art continual learning baselines, and it uses up to ten times fewer learnable parameters compared to baselines. We also make the case that sparsity is an essential ingredient for continual learning. The NISPA code is available at https://github.com/BurakGurbuz97/NISPA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. A. Rupam Mahmood
6. Mohamed Elsayed
7. Tianyu Pang
8. Yinpeng Dong

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. A. Rupam Mahmood
6. Mohamed Elsayed
7. Yinpeng Dong
8. Vamsi Krishna Ithapu

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. A. Rupam Mahmood
6. Mohamed Elsayed
7. Simone Scardapane
8. Valerio Marsocci

In Neuroscience

1. Liyuan Wang
2. Jun Zhu
3. Hang Su
4. A. Rupam Mahmood
5. Mohamed Elsayed
6. Xingxing Zhang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “PackNet [286], UCL [10], CLNP [136], AGS-CL [195] and NISPA [147] explicitly identify the important neurons or parameters for the current task and then release the unimportant parts to the following tasks, which can be achieved by iterative pruning [286], activation value [136], [147], [195], uncert”
• Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (2024, background): “Gurbuz et al. (2022) proposed using connection rewiring to induce plasticity in sparse neural networks.”