Continual World: A Robotic Benchmark For Continual Reinforcement Learning

By Maciej Wolczyk, Michal Zajkac, Razvan Pascanu, Lukasz Kuci'nski, Piotr Milo's. 2021

138 citations recorded by the source corpus.

Continual learning (CL) -- the ability to continuously learn, building on previously acquired knowledge -- is a natural requirement for long-lived autonomous reinforcement learning (RL) agents. While building such agents, one needs to balance opposing desiderata, such as constraints on capacity and compute, the ability to not catastrophically forget, and to exhibit positive transfer on new tasks. Understanding the right trade-off is conceptually and computationally challenging, which we argue has led the community to overly focus on catastrophic forgetting. In response to these issues, we advocate for the need to prioritize forward transfer and propose Continual World, a benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Meta-World as a testbed. Following an in-depth empirical evaluation of existing CL methods, we pinpoint their limitations and highlight unique algorithmic challenges in the RL setting. Our benchmark aims to provide a meaningful and computationally inexpensive challenge for the community and thus help better understand the performance of existing and future solutions. Information about the benchmark, including the open-source code, is available at https://sites.google.com/view/continualworld.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “speeding up the learning by reusing knowledge from previous tasks [12, 47].”
Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning (2022, background): “An example of such problems are robots that constantly explore and ideally learn from previous experiences (Wołczyk et al., 2021).”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…of novel tasks (Lesort et al., 2020) in various aspects including learning strategies ( e.g. , reinforcement learning (Khetarpal et al., 2018; Wołczyk et al., 2021; Xie & Finn, 2022), imitation learning (Mendez et al., 2018; Gao et al., 2021), etc. ) and task formulations ( e.g. ,…”
Building a Subspace of Policies for Scalable Continual Learning (2022, methodology): “If this information is not correctly speciﬁed, P ACK N ET is likely to fail due to either not being expressive enough to handle many tasks or being too inefﬁcient while learning only a few tasks (Wołczyk et al., 2021).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Specifically in the context of robotics, recent works have created large sets of tasks for evaluating MTL, lifelong learning, and meta-learning algorithms (Yu et al., 2019; James et al., 2020; Wołczyk et al., 2021).”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “Likewise, frameworks which introduce new models or algorithms (Lomonaco et al., 2021; Rafﬁn et al., 2019; Wolczyk et al., 2021) can also be used to create new Methods or to add new backbones to existing Methods within Sequoia.”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “For environments with continuous action spaces, task-agnostic single-headed architectures may be infeasible currently, as in (Wołczyk et al., 2021), so we leave this for future work.”