SpaceNet: Make Free Space For Continual Learning

By Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy. 2020

92 citations recorded by the source corpus.

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned tasks when the model is optimized for a new task, especially when their data is not accessible. Current architectural-based methods aim at alleviating the catastrophic forgetting problem but at the expense of expanding the capacity of the model. Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e.g. class incremental learning scenario). In this work, we propose a novel architectural-based method referred as SpaceNet for class incremental learning scenario where we utilize the available fixed capacity of the model intelligently. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. The adaptive training of the sparse connections results in sparse representations that reduce the interference between the tasks. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model, leaving space for more tasks to be learned. In particular, when SpaceNet is tested on the well-known benchmarks for CL: split MNIST, split Fashion-MNIST, and CIFAR-10/100, it outperforms regularization-based methods by a big performance gap. Moreover, it achieves better performance than architectural-based methods without model expansion and achieved comparable results with rehearsal-based methods, while offering a huge memory reduction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Pediatrics, Perinatology and Child Health

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Similar to NISPA, such approaches remove certain connections to limit interference, and they freeze essential connections to ensure stable performance on previous tasks (Golkar et al., 2019; Jung et al., 2020b; Sokar et al., 2021).”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “Another work (Sokar et al., 2020) compresses the sparse connections of each task during training, which reduce the interference between tasks and alleviate forgetting.”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, methodology): “We can carefully select and assign a subset of weights in a large network to each task (Mallya and Lazebnik 2018, Sokar et al. 2021, Fernando et al. 2017, Serra et al. 2018, Masana et al. 2021, Li et al. 2019, Yoon et al. 2018) or assign a task-speciﬁc network to each task (Rusu et al. 2016,…”