Compacting, Picking and Growing for Unforgetting Continual Learning

By Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, Chu-Song Chen. 2019

379 citations recorded by the source corpus.

Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Toalleviatethisdilemma, thenetworkarchitecturecanbe dynamically expanded if its capacity is not sufﬁcient to learn a new task well [164], [227], [228].”
Learnability and Algorithm for Continual Learning (2023, methodology): “Parameter-isolation methods train and protect a sub-network for each task (Mallya & Lazebnik, 2017; Abati et al., 2020; von Oswald et al., 2020; Rajasegaran et al., 2020b; Hung et al., 2019; Henning et al., 2021).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…Wang et al., 2022a; Saha & Roy, 2023), expansion-based methods (Rusu et al., 2016; Yoon et al., 2018; Mallya & Lazebnik, 2018; Li et al., 2019; Hung et al., 2019; Ramesh & Chaudhari, 2022; Douillard et al., 2022), or combinations thereof (Yan et al., 2021; Xie et al., 2022; Wang et al.,…”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, methodology): “Comparison with evolving architectures: Similar to progressive networks (e.g. PNN, CPG, PAE), dynamic sparse networks (e.g. CLIP, NISPA, PackNet) reduce task interference by learning a nonoverlapping task-specific sparse architecture within a fixed capacity model.”
A Theoretical Study on Solving Continual Learning (2022, background): “[54] Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and ChuSong Chen.”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “, [7, 26, 31, 9]) fix the model learnt for old tasks; and memory-based methods (e.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “, 2020), dynamic architecture-based methods (Rajasegaran et al., 2019; Hung et al., 2019; Yan et al., 2021) and memory-based methods (Rebuffi et al.”
Class Gradient Projection For Continual Learning (2022, background): “Prevalent approaches to address the catastrophic forgetting problem in continual learning can be grouped into three broad categories: regularization-based approaches [2, 22, 30, 45], expansion-based approaches [17, 25, 31, 34, 36, 40], and memory-based approaches [3, 27, 33, 38].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Many works [10,15,20,33,37] create new modules to handle the growing training distribution [26,41] dynamically.”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “However, most works in this line either focus only on the performance of the target task [16, 21, 22, 26], or assume task label is available at inference time [12,14,18,19,24,31,33].”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, methodology): “Structure-based methods [13,29] select and expand different sub-network structures involved in the optimization process of the incremental tasks.”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “Dynamic-Expansion Net (DEN) (Yoon et al., 2017) and Compacting-Picking-Growing (CPG) (Hung et al., 2019) combine the strategies of model compression/pruning, weight selection and model expansion.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “CIFAR100 is made of 50,000 train RGB images and 10,000 test RGB images of size 32 × 32 for 100 classes.”
GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “Architecture adjustments modify model architectures to incorporate CL challenges; for instance, using a recurrent neural network [13], learning overcapacitated deep neural networks in an adaptive way that repeatedly compresses sparse connections [43] or using overlapping convolutional filters [18].”
Learning with Selective Forgetting (2021, methodology): “A hybrid version of these approaches has also been proposed [ Hung et al. , 2019 ] .”