Lifelong Learning with Dynamically Expandable Networks

By Jaehong Yoon, Eunho Yang, Jeongtae Lee, Sung Ju Hwang. 2017

1,463 citations recorded by the source corpus.

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained significantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, background): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “Recent work attempted to tackle this problem either by better training algorithms [21, 25, 33, 51], structure sharing [36, 41, 49], episodic memory [9, 27, 29], machine-generated pseudo-data [18, 26, 42], or a combination of these approaches [29, 39].”
Continual Lifelong Learning with Neural Networks: A Review (2018, methodology): “e autoencoder to facilitate the addition of new MNIST digits, and it uses intrinsic replay (a generative model used for pseudo-rehearsal) to preserve the weights required to retain older information. Yoon et al., (2018) took this concept to the supervised learning paradigm and proposed a dynamically”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “To alleviate this dilemma, the network architecture can be dynamically expanded if its capacity is not sufficient to learn a new task well, such as DEN [481], CPG [174] and DGMa/DGMw [317].”
Continual and Multi-Task Architecture Search (2019, background): “Also, Yoon et al. (2018) proposed a dynamically expandable network based on in-coming new data.”
Overcoming catastrophic forgetting with hard attention to the task (2018, background): “Dynamically expandable networks (DEN; Yoon et al., 2018) also assign network capacity depending on the task at hand.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “, 2022a; Saha & Roy, 2023), expansion-based methods (Rusu et al., 2016; Yoon et al., 2018; Mallya & Lazebnik, 2018; Li et al., 2019; Hung et al., 2019; Ramesh & Chaudhari, 2022; Douillard et al., 2022), or combinations thereof (Yan et al.”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, methodology): “DEN (Yoon et al., 2018) proposed a dynamically expandable network using selective retraining, network expansion with group sparsity regularization, and neuron duplication.”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “, [24, 16, 31, 30, 26, 29]) allocate different model parameters to different tasks, fix the parameters for old tasks by masking them out during the new task learning and expand the network when needed.”
Long-Tailed Class Incremental Learning (2022, background): “Dynamically Expandable Networks (DEN) [19] performs selective retraining and dynamically expands network capacity, while Dark Experience Replay (DER) [43] dynamically expands the representation by freezing the previously-learned representation and augmenting it with additional feature dimensions fro”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, background): “Many works mitigate catastrophic forgetting during the learning process [37,14,48,68,31,42,20,2,10,3,64].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Architecture-based methods modify the network architecture dynamically to fit more tasks, by expanding the model by adding more weights [55, 52], or masking and allocating subnetworks to specific tasks [45], or by gating the parameters dynamically using a task identifier [1].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Incremental Task Learning with Incremental Rank Updates (2022, background): “Dynamic network architectures [30,19,39,38,33,7,41] try to add new neurons to the model at additional new tasks, thus the performances on previous tasks are preserved by freezing the old parameters and only updating the newly added parameters.”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “This parameter isolation is often achieved by growing new branches for tasks and freezing previous task parameters (Rusu et al., 2016; Li et al., 2021; Sahbi & Zhan, 2021; Yoon et al., 2018; Aljundi et al., 2017; Rosenfeld & Tsotsos, 2018).”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “These methods can be further categorized as expanding the model [51, 65, 26, 31, 59], or dividing the model [35, 52, 58, 17, 11].”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Many works [10,15,20,33,37] create new modules to handle the growing training distribution [26,41] dynamically.”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “The second line of work adopt dynamic models, where task-specific parameters are introduced to prevent interference among tasks, which can be achieved by growing the model [14,16,21,22,24,26,30,31,33], or introducing a gating function on the parameters [1, 18, 19].”