Continual Learning in Recurrent Neural Networks with Hypernetworks
By Benjamin Ehret, Christian Henning, Maria R. Cervera, Alexander Meulemans, Johannes von Oswald, Benjamin F. Grewe. 2020
10 citations recorded by the source corpus.
The last decade has seen a surge of interest in continual learning (CL), and a variety of methods have been developed to alleviate catastrophic forgetting. However, most prior work has focused on tasks with static data, while CL on sequential data has remained largely unexplored. Here we address this gap in two ways. First, we evaluate the performance of established CL methods when applied to recurrent neural networks (RNNs). We primarily focus on elastic weight consolidation, which is limited by a stability-plasticity trade-off, and explore the particularities of this trade-off when using sequential data. We show that high working memory requirements, but not necessarily sequence length, lead to an increased need for stability at the cost of decreased performance on subsequent tasks. Second, to overcome this limitation we employ a recent method based on hypernetworks and apply it to RNNs to address catastrophic forgetting on sequential data. By generating the weights of a main RNN in a task-dependent manner, our approach disentangles stability and plasticity, and outperforms alternative methods in a range of experiments. Overall, our work provides several key insights on the differences between CL in feedforward networks and in RNNs, while offering a novel solution to effectively tackle CL on sequential data.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Guillaume Hennequin
- Ta-Chu Kao
- Kristopher T. Jensen
- Gido M. van de Ven
- Alberto Bernacchia
- Andreas S. Tolias
- Lawrence Carin
- Johannes von Oswald
In Computer Vision and Pattern Recognition
- Guillaume Hennequin
- Ta-Chu Kao
- Kristopher T. Jensen
- Alberto Bernacchia
- Gido M. van de Ven
- Andreas S. Tolias
- Lawrence Carin
- Johannes von Oswald
Citations in the corpus
- Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “…to previous methods in recurrent neural networks (RNNs), a setting that has recently proven challenging for continual learning (Duncker et al., 2020; Ehret et al., 2020) and which is of interest to the study of continual learning in biological circuits (Duncker et al., 2020; Yang et al., 2019).”