Selective Experience Replay for Lifelong Learning

By David Isele, Akansel Cosgun. 2018

543 citations recorded by the source corpus.

Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial---when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “rameters. Recent work attempted to tackle this problem either by better training algorithms [21, 25, 33, 51], structure sharing [36, 41, 49], episodic memory [9, 27, 29], machinegenerated pseudo-data [18, 26, 42], or a combination of these approaches [29, 39]. Benchmarks to compare these methods typ”
Experience Replay for Continual Learning (2018, background): “A noteworthy use of experience replay buffers to protect against catastrophic forgetting was demonstrated in Isele and Cosgun [10] on toy tasks, with a focus on how buffers can be made smaller.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “An empirical study [181] evaluates different principles of selecting old experiences for replay, with matching training distributions usually performing best.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Isele & Cosgun (2018) observed the performance of agent was strongest when the distribution of data in the replay buffer matches the test distribution.”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “…ones that allocate more neural resources over time (Ring, 1997; Rusu et al., 2016), and ones that explicitly store (or train generative models to mimic) past data which are used in various ways to improve the memory of the network (Isele & Cosgun, 2018; Sprechmann et al., 2018; Shin et al., 2017).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “More recently, based on Isele & Cosgun (2018); Hayes & Kanan (2021), Araujo et al. (2022) further included three more basic methods, and compared the seven methods for natural language processing applications.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “These methods store (Rebufﬁ et al., 2017; Rolnick et al., 2019; Isele & Cosgun, 2018) or generate (Shin et al., 2017; Atkinson et al., 2018; Ramapuram et al., 2020) examples of previous tasks.”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “We set the replay buffer capacity to 5000 dialogues and use reservoir sampling (Isele and Cosgun, 2018) when the buffer is full.”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “For continual reinforcement learning, Isele & Cosgun (2018) propose selection strategies based on prediction errors or rewards, but these do not promote diversities within the memory.”
Meta-attention for ViT-backed Continual Learning (2022, background): “Existing continual learning methods can be broadly categorized into three schools: replay methods [3,6,19,36,37], regularization methods [21,23, 26, 27, 34, 50, 52] and mask methods [18, 30–32, 41].”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, background): “It is a global distribution matching strategy that ensures that at any given time the distribution of samples in the buffer will approximately match the distribution of all the samples seen so far (Isele & Cosgun, 2018).”
CoMPS: Continual Meta Policy Search (2021, methodology): “…at a time, as in the home cleaning robot example before, it still relies on storing experience from prior tasks, similar to rehearsal-based methods (Isele & Cosgun, 2018; Riemer et al., 2018; Rolnick et al., 2019; Atkinson et al., 2021), but instead uses this information to accelerate sequential…”
Continual Learning via Local Module Composition (2021, methodology): “Continual learning methods typically address the problem of forgetting through parameter regularization [46, 70, 99], replay [89, 78, 3, 73, 54, 12, 97, 39, 81, 13] or dynamic architectures (and MoEs) [83, 87, 52, 85, 53, 41].”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “When the storage space is limited, it is important to ﬁnd a suitable subset of data that can approximate the entire data distribution, e.g., SER [17] focuses on exemplar selection techniques.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, methodology): “Replay methods [34, 20, 28] utilize data samples in the previous session as either the network inputs or the constraints for optimization in the new session.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “Different selective experience replay strategies can be used for preserving performance on past tasks (Isele and Cosgun 2018).”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Isele and Cosgun (2018) compared four experience selection strategies to augment a first-in first-out queue: TD error, absolute reward, distribution matching based on reservoir sampling, and state-space coverage maximization based on the nearest neighbors to an experience.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “All our methods use prioritized replay buffer that resembles the surprise strategy (Isele & Cosgun, 2018).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “As a result, replay has become a very successful approach for tackling continual RL (Riemer et al., 2019; Isele and Cosgun, 2018; Rolnick, Ahuja, Schwarz, Lillicrap, and Wayne, 2019; Oh, Shin, Yang, and Hwang, 2021; Henning, Cervera, D’Angelo, Von Oswald, Traber, Ehret, Kobayashi, Grewe, and…”