Continual learning with hypernetworks

By Johannes von Oswald, Christian Henning, João Sacramento, Benjamin F. Grewe. 2019

428 citations recorded by the source corpus.

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Subfields: Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “In a broader sense, sampling parameters from taskconditioned parameter distributions (MERLIN [221], PR [160], PGMA [169], HNET [420]), as well as stabilizing important parameters with weight regularization, can be seen as a form of deriving task-specific/adaptive parameters.”
Learnability and Algorithm for Continual Learning (2023, methodology): “CCG (Abati et al., 2020), Expert Gate (Aljundi et al., 2017), HyperNet (von Oswald et al., 2020) and PREnt (Henning et al., 2021) either build a separate network or use entropy to predict the task-id.”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “For example, expanding the capacity of a nonlinear model like an MLP module, transformer attention head [13], or hypernetwork [63] would alter the weight outputs of previously learned components.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “Finally, for parameter isolation, we use CCG [63], HyperNet [55], HAT [3], SupSup [4] (Sup), and PR [65].”
Incremental Task Learning with Incremental Rank Updates (2022, background): “[24] also proposes task-conditioned hypernetworks for continual learning.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “Related work includes (Shin et al. 2017; Wu et al. 2018; Kamra, Gupta, and Liu 2017; Seff et al. 2017; Hu et al. 2019; Ostapenko et al. 2019; Hayes et al. 2019; von Oswald et al. 2020).”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “…have been devoted to develop better continual learning algorithms, ranging from the dynamic architectures (Rusu et al., 2016; Serra et al., 2018; von Oswald et al., 2020; Yoon et al., 2018; Li et al., 2019; Xu & Zhu, 2018), experience replay (Lin, 1992; Chaudhry et al., 2019b; Aljundi et al.,…”
Continual Learning with Recursive Gradient Optimization (2022, background): “…2018; Kaushik et al., 2021), add additional components as new tasks arrive (Rusu et al., 2016; Fernando et al., 2017; Alet et al., 2018; Chang et al., 2019; Li et al., 2019), or use larger networks to generate network parameters (Aljundi et al., 2017; Yoon et al., 2019; von Oswald et al., 2019).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Oswald et al. [120] proposed to store a task embedding for each task in the buffer.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Experience replay with stored data [2, 3, 8, 9, 15, 19, 24, 29, 40, 46, 47, 48, 59] or samples from a generative model [27, 28, 45, 51, 56] is highly effective when storing training data or training/saving a generative model is possible, unlike the DFCIL setting.”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “While we enforce this stringent condition in this paper, our approach may be easily combined with memory-based techniques such as coresets or generative replay [8, 10, 13, 31, 33, 35, 37, 41, 42, 44, 47].”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, background): “Others use different functions as components in the network, either Hypernetworks [39], Deep Artificial Neurons (DANs) [40], or Compositional Structures [41], so that the network components are more flexible during learning.”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “Table 2: Accuracy results for 1 class per task for PCL and all baselines except HNET as it does not work with one class per task.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…al., 2018; Ostapenko et al., 2019; Abati et al., 2020; Liu, Su, Liu, Schiele, & Sun, 2020; Titsias, Schwarz, de G. Matthews, Pascanu, & Teh, 2020; von Oswald, Henning, Sacramento, & Grewe, 2020; Ye & Bors, 2020) or midlevel CNN feature representations (van de Ven et al., 2020; Lao, Jiang,…”
Rectification-based Knowledge Retention for Continual Learning (2021, methodology): “Split CIFAR-10/100: For the split CIFAR-10/100 tasks we use the ResNet-32 architecture and compare our method RKR with CCLL and HNET [28], which are the state-ofthe-art methods for this setup and use task labels during testing.”