Continual Reinforcement Learning with Multi-Timescale Replay

By Christos Kaplanis, Claudia Clopath, Murray Shanahan. 2020

18 citations recorded by the source corpus.

In this paper, we propose a multi-timescale replay (MTR) buffer for improving continual learning in RL agents faced with environments that are changing continuously over time at timescales that are unknown to the agent. The basic MTR buffer comprises a cascade of sub-buffers that accumulate experiences at different timescales, enabling the agent to improve the trade-off between adaptation to new data and retention of old knowledge. We also combine the MTR framework with invariant risk minimization, with the idea of encouraging the agent to learn a policy that is robust across the various environments it encounters over time. The MTR methods are evaluated in three different continual learning settings on two continuous control tasks and, in many cases, show improvement over the baselines.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “MTR [201] employs a cascade of interacting sub-buffers to accumulate experiences at different timescales.”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “However, most methods in one field can be instantiated in the other, resulting in duplicate efforts such as replay for CSL [51, 34, 58, 35, 49] and replay for CRL [63, 54, 25].”