Continual Learning via Neural Pruning

By Siavash Golkar, Michael Kagan, Kyunghyun Cho. 2019

184 citations recorded by the source corpus.

We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.

Subfields: Artificial Intelligence, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, neutral): “…learning (Lake et al., 2011; Romera-Paredes and Torr, 2015; Vinyals et al., 2016; Pan and Yang, 2010), meta-learning and life-long learning (Thrun and Mitchell, 1995; Silver et al., 2013; Chen and Liu, 2016; Parisi et al., 2018; Golkar et al., 2019; Sodhani et al., 2018; Lopez-Paz et al., 2017).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “PackNet [224], UCL [115], CLNP [225] and AGS-CL [116] explicitly identify the important neurons or parameters for the current task and then release the unimportant parts to the following tasks, which can be achieved by iterative pruning [224], activation value [116], [225], [226], uncertainty…”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, neutral): “By contrast, several other methods (e.g. NISPA (Gurbuz & Dovrolis, 2022), CLNP (Golkar et al., 2019), PackNet (Mallya & Lazebnik, 2018), PAE (Hung et al., 2019b)) proposed dynamic sparse networks based on neuronal model sparsification with fixed model capacity.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, background): “As presented in the main paper, there exist some expansion-based continual learning methods like [69, 34, 17, 59, 25, 19, 39].”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “First, NISPA matches or outperforms CLNP on EF-MNIST, and it does much better than other baselines although they have ﬁve times more learnable parameters.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Many works [10,15,20,33,37] create new modules to handle the growing training distribution [26,41] dynamically.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “CIFAR100 is made of 50,000 train RGB images and 10,000 test RGB images of size 32 × 32 for 100 classes.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “On a model level, a combination of CL with Bayesian principles should allow to identify better the importance of each parameter of an NN and aid parameter pruning and quantization (e.g. Ebrahimi et al., 2020; Golkar et al., 2019).”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Other works focus on adaptive architectures that can extend themselves to integrate new classes [67, 40] or dynamically re-arrange coexisting sub-networks [20] each specialized in one specific task [19, 23, 32], or to explicitly correct the classifier drift [66, 74, 3, 4] that happens with continual”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Other solutions judiciously add new capacity only when absolutely needed [36 – 38], expand more aggressively and then prune or compress parts of the model [39,40], or channel learning of new tasks to unused parts of a large ﬁ xed model [41 – 45].”
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, methodology): “…on developing methods for mitigating forgetting, with techniques to implicitly or explicitly partition parameters for diﬀerent tasks [14, 25, 26, 27], regularization (elasticity) and Bayesian approaches [6, 15, 28, 29, 18, 30] as well as directly storing data from earlier tasks in replay buﬀers…”
Supermasks in Superposition (2020, methodology): “In [31, 11], the weights of disjoint subnetworks are trained for each new task.”
CPR: Classifier-Projection Regularization for Continual Learning (2020, background): “…et al., 2017; Zenke et al., 2017; Nguyen et al., 2018; Ahn et al., 2019; Aljundi et al., 2019), memory replay (Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Rebuffi et al., 2017; Kemker & Kanan, 2018), and dynamic network architecture (Rusu et al., 2016; Yoon et al., 2018; Golkar et al., 2019).”
Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, methodology): “[10] developed a notion of active and inactive nodes and implemented pruning and freezing schemes, which are similar to ours, but they required several hyperparameters and involved cumbersome, non-adaptive threshold tuning steps that require a separate validation set.”
Bayesian Structure Adaptation for Continual Learning (2019, methodology): “Among these, strong evidence in support of our assumptions can be found in (Golkar et al., 2019), which also learns different sparse subsets of the weights of each layer of the network for different tasks.”
Continual Unsupervised Representation Learning (2019, background): “Expansion-based models dynamically increase capacity to allow for additional tasks (Rusu et al., 2016; Yoon et al., 2017; Draelos et al., 2017), and optionally prune the network to constrain capacity (Zhou et al., 2012; Golkar et al., 2019).”
Improving and Understanding Variational Continual Learning (2019, methodology): “This pruning effect is also similar to that considered in Golkar et al. [6], where they prune out entire units.”
Task Agnostic Continual Learning via Meta Learning (2019, methodology): “In (Golkar et al., 2019), pruning techniques were applied to minimize the growth of the model after each task.”