Experience Replay for Continual Learning
By David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, Greg Wayne. Neural Information Processing Systems, 2018
1,717 citations recorded by the source corpus.
Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.
Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning, Meta & Continual Learning
View Paper: https://arxiv.org/abs/1811.11682
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Dacheng Tao
- Jianfeng Gao
- Quoc V. Le
- Trevor Darrell
- Ruslan Salakhutdinov
In Computer Science Applications
In Management Science and Operations Research
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Yoshua Bengio
- Josh Tenenbaum
- Maosong Sun
- Behnam Neyshabur
- Zhiyuan Liu
In Reinforcement Learning
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Joelle Pineau
- Shimon Whiteson
- Abhinav Kumar Gupta
- Igor Mordatch
- Razvan Pascanu
In Meta & Continual Learning
- Tinne Tuytelaars
- Jorge Armando Mendez Mendez
- Zixuan Ke
- Eric Eaton
- Bing Liu
- Han-Jia Ye
- Fei Ye
- Adrian G. Bors
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “CLEAR [364] mixes off-policy learning from old experiences and onpolicy learning from novel experiences, with behavioral cloning to further promote stability.”
- Policy Consolidation for Continual Reinforcement Learning (2019, background): “Recently it has been shown that simply maintaining a uniform distribution of the data over the life-time of the model yields decent results (Isele & Cosgun, 2018; Rolnick et al., 2018).”
- Learnability and Algorithm for Continual Learning (2023, methodology): “…the current data and the saved buffer data (Rusu et al., 2016; LopezPaz & Ranzato, 2017; Chaudhry et al., 2019a; Hou et al., 2019; Wu et al., 2019; Rolnick et al., 2019; Buzzega et al., 2020; Rajasegaran et al., 2020a; Liu et al., 2021; Cha et al., 2021; Yan et al., 2021; Wang et al., 2022b; Guo…”
- Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Replay often stores a subset of the learning task data within the memory to replay during incremental learning tasks, such as experience replay [Rolnick et al., 2019].”
- Progressive Prompts: Continual Learning for Language Models (2023, background): “While replay-based approaches have shown strong results for classification, question answering and relation extraction tasks, they pose significant memory requirements due to storing large number of samples for rehearsal (Rolnick et al., 2019; Huang et al., 2021).”
- A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “, Experience Replay (ER) [41], iCaRL [39] and GDumb [36].”
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Rehearsal with stored data [3–5, 7, 9, 10, 18, 19, 24, 29, 42, 51–53, 64, 68] or samples from a generative model [27, 28,48,56,60] is highly effective when storing training data or training/saving a generative model is possible.”
- A Theoretical Study on Solving Continual Learning (2022, background): “Memorizing some old examples and using them to jointly train the new task is another popular approach (called replay ) [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
- Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “Behavioral cloning achieves more promising results [44, 35].”
- Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
- Incremental Task Learning with Incremental Rank Updates (2022, background): “Incremental task learning is a subcategory of continual learning or lifelong learning approaches aim to address the problem of catastrophic forgetting by adapting the network or training process to learn new tasks without forgetting the previously learned ones [23,16,3,2,4,6,28,29,12].”
- Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “Many systems employ this approach, e.g., iCaRL (Rebuffi et al. 2017), GEM (Lopez-Paz and Ranzato 2017), A-GEM (Chaudhry et al. 2019), RPSnet (Rajasegaran et al. 2019) and others (Rusu et al. 2016; Wu et al. 2019; Rolnick et al. 2019b; de Masson d’Autume et al. 2019; Hou et al. 2019).”
- NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “These methods store (Rebuffi et al., 2017; Rolnick et al., 2019; Isele & Cosgun, 2018) or generate (Shin et al.”
- Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “Although several methods effective at reducing forgetting when learning new tasks were proposed (Kirkpatrick et al., 2017; Lee et al., 2017; Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Masse et al., 2018; Liu et al., 2018; Rolnick et al.,”
- Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “• the first framework for continual RL to optimise the dialogue policy of a task-oriented dialogue system, two baseline architectures, an implementation of the state-of-the-art continual RL algorithm (Rolnick et al., 2018) and continual learning metrics for evaluation based on Powers et al.”
- Information-theoretic Online Memory Selection for Continual Learning (2022, positive): “Specifically, replay-based methods (Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017; Rolnick et al., 2019) tackle the continual learning problem by keeping a replay memory for rehearsals over the past data.”
- Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, neutral): “…et al., 2018; Li et al., 2019; Xu & Zhu, 2018), experience replay (Lin, 1992; Chaudhry et al., 2019b; Aljundi et al., 2019a; Riemer et al., 2019; Rolnick et al., 2019; Wu et al., 2019), regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Ritter et al., 2018), to…”
- Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “[29] introduces experience replay to continual learning.”
- The Challenges of Continuous Self-Supervised Learning (2022, background): “Rehearsal techniques [2, 6, 60, 67, 68, 73] store and replay a small set of training samples from previous tasks to avoid forgetting previously learned skills or concepts.”