Towards Continual Reinforcement Learning: A Review and Perspectives

By Khimya Khetarpal, Matthew Riemer, Irina Rish, Doina Precup. 2020

437 citations recorded by the source corpus.

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

Subfields: Artificial Intelligence, Public Health, Environmental and Occupational Health, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Public Health, Environmental and Occupational Health

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “In contrast, the latest surveystypicallycollateonlyalocalaspectofcontinuallearning, withrespecttoitsbiologicalunderpinnings[1],[3],[9],[10], specializedsettingsforvisualclassiﬁcation[11],[12],[13],[14], andspeciﬁcextensionsforNLP[15],[16]orRL[17].”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “The ability of continual learning (CL) systems ([17, 22]) to utilize knowledge from previously seen tasks in order to maximize transfer on the current task is a significant challenge for the field.”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “Despite progress in continual learning (Lange et al., 2019; Parisi et al., 2019; Biesialska et al., 2020; Khetarpal et al., 2020a; Hadsell et al., 2020), there is – to the best of our knowledge – no work that addresses continual reinforcement learning (continual RL) of the dialogue policy, even…”
CoMPS: Continual Meta Policy Search (2021, background): “Continual or online learning studies the streaming data setting, where experience is used for training as soon as it is received (Thrun & Pratt, 2012; Hazan et al., 2016; Chen & Liu, 2016; Parisi et al., 2019; Khetarpal et al., 2020).”
Continual Learning via Local Module Composition (2021, background): “The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “All rights reserved. et al. 1994; Khetarpal et al. 2020), a paradigm where agents train on tasks sequentially, while seeking to maintain performance on previously mastered tasks (Kirkpatrick et al. 2016; Schwarz et al. 2018; Rolnick et al. 2019).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “The possibility of applying CL methods in reinforcement learning scenarios has been explored for a long time, see [24] for a recent review.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Designing overly-tailored experiments for a specific lifelong RL problem can entail unwanted bias in the study (Khetarpal et al., 2020).”
Avalanche: an End-to-End Library for Continual Learning (2021, neutral): “The Evaluator object directly interacting with the algorithm provides a uniﬁed interface to control and compute several performance metrics ( p i ), delegating results logging to the Logger(s) objects. to Robotics [28, 40, 38], from Reinforcement Learning [25, 31] to Sequence Learning [8], among oth”
NovelGridworlds: A Benchmark Environment for Detecting and Adapting to Novelties in Open Worlds (2021, methodology): “There are increasing efforts for developing AI agents to adapt to novelties in open worlds, using techniques such as reinforcement learning (RL) [7, 12], planning-based approaches [8, 9], and hybrid approaches [14, 16].”
Continuum: Simple Management of Complex Continual Learning Scenarios (2021, background): “Continual learning is therefore naturally adapted to this learning paradigm [21].”
Biological underpinnings for lifelong learning machines (2022, background): “It should be noted that there is also a body of artificial intelligence (AI) research that tackles the lifelong learning problem from a less clearly biological perspective 2–10 .”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “In recent years, there has been a growing interest in building agents that continuously learn skills or tasks without forgetting previously learned behavior (Khetarpal et al., 2020).”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “Deep RL is essentially a CL problem due to its learning mode of exploring while learning [20], and it is particularly vulnerable to catastrophic interference, even within many single-task settings where the environment is stationary (such as Atari 2600 games or even simpler classic control tasks in…”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, background): “In the literature, this is referred to as continual learning (Biesialska et al., 2020; Khetarpal et al., 2020; Hadsell et al., 2020).”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, methodology): “Since we can not list all related methods here, we encourage readers to check recent surveys (Khetarpal et al. 2020, Delange et al. 2021).”
Continual Learning In Environments With Polynomial Mixing Times (2021, background): “In contrast, the average reward per step objective, which explicitly includes an average over the stationary distribution, avoids inducing myopic biases and hence is well-suited for continual RL problems [5, 1].”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “In LRL (see Khetarpal et al. (2020) for a survey), each task is a Markov decision process (MDP), and the agent must learn a policy for interacting with its environment to maximize its lifetime performance (e.g., av- ar X iv :2 20 8.”