DisCoRL: Continual Reinforcement Learning via Policy Distillation

By Kalifou René Traoré, Hugo Caselles-Dupré, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz Rodríguez, David Filliat. 2019

66 citations recorded by the source corpus.

In multi-task reinforcement learning there are two main challenges: at training time, the ability to learn different policies with a single model; at test time, inferring which of those policies applying without an external signal. In the case of continual reinforcement learning a third challenge arises: learning tasks sequentially without forgetting the previous ones. In this paper, we tackle these challenges by proposing DisCoRL, an approach combining state representation learning and policy distillation. We experiment on a sequence of three simulated 2D navigation tasks with a 3 wheel omni-directional robot. Moreover, we tested our approach's robustness by transferring the final policy into a real life setting. The policy can solve all tasks and automatically infer which one to run.

Subfields: Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “Behavioral cloning achieves more promising results [44, 35].”
Continual Learning for Robotics (2019, methodology): “However, many approaches [160, 68, 6, 13] also tackle use cases on real robots.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “…to be very versatile and applicable in large set of scenarios (Kalifou et al., 2019; Diethe et al., 2018; Lesort et al., 2019; Prabhu et al., 2020; Traoré et al., 2019; Belouadah and Popescu, 2018; Wu et al., 2019; Hou et al., 2019; Lesort, 2020; Douillard et al., 2020a) in comparison with other…”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “Catastrophic interference within the single-task RL is a special case of CRL, which involves not only the strategies to mitigate interference but also the context detection and identiﬁcation techniques.”
Continuum: Simple Management of Complex Continual Learning Scenarios (2021, background): “) [22, 19] or tasks [20, 43], (incremental learning) either the current environment change while learning a single tasks (lifelong learning).”
Biological underpinnings for lifelong learning machines (2022, background): “For example, a system that needs to remember (for example, in a database) all experiences of its past will require ever-increasing storage capacity (for example, in replay buffers), although there are attempts to compress what needs to be stored across longer timescales 30–32 .”
Acquisition of chess knowledge in AlphaZero (2021, background): “Representation learning in RL agents [45, 46, 47, 48, 49, 50] can be beneficial when used to learn low-dimensional representations for states, policies and actions, hoping to meaningfully capture the variations in agent’s environment and disentangle the contributing factors.”
Explainability in Deep Reinforcement Learning (2020, background): “Moreover, when encompassing human advice during training, it can be derived into advisable learning which further improves user trust as the model can understand human natural language and yields clear and precise explanations [100].”
Building a Subspace of Policies for Scalable Continual Learning (2022, background): “Another class of methods employs knowledge distillation to improve transfer in CRL [17, 37, 23, 40, 5, 20, 46].”