Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting

By Craig Atkinson, Brendan McCane, Lech Szymanski, Anthony V. Robins. 2018

121 citations recorded by the source corpus.

Neural networks can achieve excellent results in a wide variety of applications. However, when they attempt to sequentially learn, they tend to learn the new task while catastrophically forgetting previous ones. We propose a model that overcomes catastrophic forgetting in sequential reinforcement learning by combining ideas from continual learning in both the image classification domain and the reinforcement learning domain. This model features a dual memory system which separates continual learning from reinforcement learning and a pseudo-rehearsal system that "recalls" items representative of previous tasks via a deep generative network. Our model sequentially learns Atari 2600 games while continuing to perform above human level and equally well as independent models trained separately on each game. This result is achieved without: demanding additional storage requirements as the number of tasks increases, storing raw data or revisiting past tasks. In comparison, previous state-of-the-art solutions are substantially more vulnerable to forgetting on these complex deep reinforcement learning tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “Although our algorithm is intended to operate in settings where new tasks are revealed sequentially, one at a time, as in the home cleaning robot example before, it still relies on storing experience from prior tasks, similar to rehearsal-based methods (Isele & Cosgun, 2018; Riemer et al., 2018; Rol”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Experience replay has also been widely used in reinforcement learning (Mnih et al., 2015, 2013; Van Hasselt, Guez, & Silver, 2016; Lillicrap et al., 2016; Lin, 1992; Adam, Busoniu, & Babuska, 2011; Foerster et al., 2017; Kapturowski, Ostrovski, Dabney, Quan, & Munos, 2019; Atkinson et al., 2021).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This has led some to explore replacing replay buffers with pseudo-rehearsals sampled from a generative model (Robins, 1995; Atkinson et al., 2018; Daniels et al., 2022).”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “However, Atkinson et al. (2021) uses GANs in the raw perceptual space of the agent (aka veridical replay Hayes et al. (2021)).”
Building a Subspace of Policies for Scalable Continual Learning (2022, background): “There is also a large body of work which leverages the shared structure of the tasks [50, 30, 25, 27, 1, 43], meta-learning [19, 44, 4, 8, 33, 52, 39], or generative models [35, 42, 41, 2] to improve CRL agents, but these are less related to CSP and not very competitive.”
Towards Knowledgeable Supervised Lifelong Learning Systems (2020, methodology): “Pseudo-recursal (Atkinson et al., 2018a) used Generative Adversarial Networks (GAN) (Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, & Bengio, 2014) to generate examples from previous tasks used to retrain old tasks while learning a new task.”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, background): “The student-teacher dual memory system is also exploited in multi-task RL, inducing behavioral cloning between the current network (the student network) and its old version (the teacher network) (Rolnick et al. 2019, Kaplanis et al. 2019, Atkinson et al. 2021).”