On Training Recurrent Neural Networks for Lifelong Learning

By Shagun Sodhani, A. P. Sarath Chandar, Yoshua Bengio. 2018

19 citations recorded by the source corpus.

Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty. To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task. As a step towards developing true lifelong learning systems, we unify Gradient Episodic Memory (a catastrophic forgetting alleviation approach) and Net2Net(a capacity expansion approach). Both these models are proposed in the context of feedforward networks and we evaluate the feasibility of using them for recurrent networks. Evaluation on the proposed benchmark shows that the unified model is more suitable than the constituent models for lifelong learning setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, neutral): “…learning (Lake et al., 2011; Romera-Paredes and Torr, 2015; Vinyals et al., 2016; Pan and Yang, 2010), meta-learning and life-long learning (Thrun and Mitchell, 1995; Silver et al., 2013; Chen and Liu, 2016; Parisi et al., 2018; Golkar et al., 2019; Sodhani et al., 2018; Lopez-Paz et al., 2017).”
LAMOL: LAnguage MOdeling for Lifelong Language Learning (2019, neutral): “Towards Training Recurrent Neural Networks for Lifelong Learning (Sodhani et al., 2018) uniﬁes Gradient episodic memory (Lopez-Paz et al., 2017) and Net2Net (Chen et al., 2015a).”
Continual Learning for Robotics (2019, background): “However, only few datasets, such as CORe50 (Lomonaco and Maltoni, 2017) or (Sodhani et al., 2018), are specifically built with continual learning in mind.”
Continual Representation Learning for Biometric Identification (2020, neutral): “Continual learning is also named life-long learning [46, 55, 57], incremental learning [7, 48, 54] and sequential learning [5, 10] in previous works.”
Continual Learning with Gated Incremental Memories for sequential data processing (2020, background): “the standard approach of CL literature [23, 24] by dividing each dataset into groups of non-overlapping classes, which we call subtasks.”
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text (2019, background): “Not surprisingly, the performance of all models degrades monotonically as we increase the length of the test clauses, which highlights the challenge of “zero-shot” systematic generalization (Lake and Baroni, 2018; Sodhani et al., 2018).”
Neural Architecture Search for Class-incremental Learning (2019, methodology): “These methods are susceptible to the phenomenon of capacity saturation (Sodhani, Chandar, and Bengio 2018), where a neural network’s ability to generalize to new concepts is limited by its fixed capacity.”
TAG: Task-based Accumulated Gradients for Lifelong learning (2021, methodology): “Another closely related methods, called Expansionbased methods, handle the lifelong learning problem by expanding the model capacity in order to adapt to new tasks [53, 44].”