Toward Training Recurrent Neural Networks for Lifelong Learning

By Shagun Sodhani, Sarath Chandar, Yoshua Bengio. 2019

101 citations recorded by the source corpus.

Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty. To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task. As a step toward developing true lifelong learning systems, we unify gradient episodic memory (a catastrophic forgetting alleviation approach) and Net2Net (a capacity expansion approach). Both models are proposed in the context of feedforward networks, and we evaluate the feasibility of using them for recurrent networks. Evaluation on the proposed benchmark shows that the unified model is more suitable than the constituent models for lifelong learning setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “…the catastrophic forgetting phenomenon and can be categorized into three groups: ( Ranzato, 2017; Chaudhry et al., 2018; Wang et al., 2020) (3) network expansion-based approaches dynamically expand the network based upon new tasks (Rusu et al., 2016; Aljundi et al., 2017; Sodhani et al., 2020).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Sodhani et al. [112] proposed to unify GEM and Net2Net [113] frameworks to enable the model to mitigate catastrophic forgetting and increase its capacity.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, positive): “…of supervised learning literature has focused on over-coming forgetting in feedforward or convolutional neural networks, there has also been work focused on using replay to mitigate forgetting in recurrent neural networks (Parisi, Tani, Weber, and Wermter, 2018; Sodhani, Chandar, & Bengio, 2020).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “In [132], the authors unify GEM [13] and Net2Net [135] to tackle forgetting in RNN.”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Moreover, humans can continually learn and accumulate knowledge over their lifetime, while the current learning algorithms are known to suffer from several challenges when training over a sequence of tasks [33, 15, 8, 45].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “Importantly, if tasks are related, then knowledge transfer between tasks should lead to a better generalization and faster learning (Lopez-Paz and Ranzato, 2017; Sodhani et al., 2020).”
Knowledge Capture and Replay for Continual Learning (2020, background): “Also, KD techniques do not satisfy the critical requirements for CL [Sodhani et al., 2020].”
Efficient Meta Lifelong-Learning with Limited Memory (2020, methodology): “While prior methods use memory to deﬁne optimization constraints (Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019; Sodhani et al., 2020), recent work use either stored examples (Sprech-mann et al., 2018) or generated synthetic data (Shin et al., 2017; Sun et al., 2020) to perform experience…”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “A Net2Net technique was used for expanding the RNN in (Sodhani et al., 2019).”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “A Net2Net technique [97] was combined with gradient episodic memory [98].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “For now, we will start with a typical categorization of existing deep continual works into the three categories of regularization, rehearsal and architectural approaches, in consistency ∙ LWF (LiandHoiem,2016) ∙ EBLL (Rannenetal.,2017) ∙ Pseudorehearsal (Robins,1995) ∙ Pseudo-recurrentnets (French,1”
VariGrow: Variational Architecture Growing for Task-Agnostic Continual Learning based on Bayesian Novelty (2022, background): “While regularization-and memory-based methods focus on retaining the knowledge learned from the old tasks, expansion-based methods lean towards better absorbing new knowledge and circumvent the capability saturation (Sodhani et al., 2020).”
Continual Learning in Recurrent Neural Networks with Hypernetworks (2020, background): “[32] Shagun Sodhani, Sarath Chandar, and Yoshua Bengio.”
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “This can also be seen in several reviews [4,5,11,29,30,35] reporting few approaches that consider architectures besides CNNs, despite the attempts to investigate CL in RNNs [9,43].”
Organizing recurrent network dynamics by task-computation to enable continual learning (2020, background): “Overcoming these challenges has been the subject of continual learning [11–15].”