Policy Consolidation for Continual Reinforcement Learning

By Christos Kaplanis, Murray Shanahan, Claudia Clopath. 2019

61 citations recorded by the source corpus.

We propose a method for tackling catastrophic forgetting in deep reinforcement learning that is \textit{agnostic} to the timescale of changes in the distribution of experiences, does not require knowledge of task boundaries, and can adapt in \textit{continuously} changing environments. In our \textit{policy consolidation} model, the policy network interacts with a cascade of hidden networks that simultaneously remember the agent's policy at a range of timescales and regularise the current policy by its own history, thereby improving its ability to learn without forgetting. We find that the model improves continual learning relative to baselines on a number of continuous control tasks in single-task, alternating two-task, and multi-agent competitive self-play settings.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Experience Replay for Continual Learning (2018, background): “Reducing this deviation is our motivation for the behavioral cloning component of CLEAR, inspired by work showing the power of policy consolidation [12], self-imitation [23], and knowledge distillation [5].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, background): “…2018), Diffusion-based Neuromodulation (Velez and Clune, 2017), IMM (Lee et al., 2017), Expandable Networks (Yoon et al., 2018; Li et al., 2019), RPSNet (Rajasegaran et al., 2019), reinforcement learning (Kaplanis et al., 2019; Rolnick et al., 2019), and meta-learning (Javed and White, 2019).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “[23] Christos Kaplanis, Murray Shanahan, and Claudia Clopath.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “RL can be a natural ﬁt for studying LLL as it provides an agent-environment interaction paradigm wherein the agent is exposed to non-stationary streams of data (Kaplanis et al., 2018; 2019).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2015) from past tasks when learning a new task to prevent catastrophic forgetting as in (Rusu et al., 2015; Li and Hoiem, 2016; Riemer et al., 2016; Espeholt et al., 2018; Schwarz et al., 2018; Berseth et al., 2018; Kaplanis et al., 2019; Traoré et al., 2019; Tirumala et al., 2019; Igl et al., 202”
Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “It also aims to work in scenarios where task boundaries are not well defined or provided, or when the data distribution shifts slowly or when a multi-task solution does not exist (Rao et al., 2019; Aljundi et al., 2019; He et al., 2019; Kaplanis et al., 2019).”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “Continual Learning as an emerging field in Artificial Intelligence is connected to many other areas such as Meta Learning [4, 35, 24, 62], Few Shot Learning [75, 20], Multi-task and Transfer Learning [24, 35] and the closely related problem of exploring task boundary or task detection [60, 2, 25, 36”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “…of memory to mitigate catastrophic forgetting in deep RL include [Kaplanis et al., 2018], which consolidates the individual parameters at multiple timescales, and [Kaplanis et al., 2019], which consolidates the agent’s policy at multiple timescales, using a cascade of hidden policy networks.”
Building a Subspace of Policies for Scalable Continual Learning (2022, background): “…models that struggle to learn a large number of diverse behaviors (Hinton et al., 2006; Rusu et al., 2016a; Li & Hoiem, 2018; Bengio & LeCun, 2007; Kaplanis et al., 2019; Traoré et al., 2019; Kirkpatrick et al., 2017; Schwarz et al., 2018; Mallya & Lazebnik, 2018), or growing-size models that…”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, background): “The student-teacher dual memory system is also exploited in multi-task RL, inducing behavioral cloning between the current network (the student network) and its old version (the teacher network) (Rolnick et al. 2019, Kaplanis et al. 2019, Atkinson et al. 2021).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Kaplanis et al. (2019) trained a similar monolithic architecture in the task-agnostic setting, including for tasks with continuous distribution shifts.”