Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

By Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone. Journal of machine learning research, 2020

748 citations recorded by the source corpus.

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/e102cd42c402026a3862d2e60a75eed7c78860a2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Replay-Guided Adversarial Environment Design (2021, background): “In inducing parallel curricula, DCD follows a rich lineage of curriculum learning methods [2, 32, 28, 23].”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, neutral): “As such, we consider the design of lifelong learning algorithms to be an open area of research in LLDM. (T4) Robustness to Task Ordering It is well-known that task curriculum inﬂuences policy learning [6, 48].”
Structural Similarity for Improved Transfer in Reinforcement Learning (2022, neutral): “One potential solution to both of these problems is to leverage transfer learning [5, 15, 29], i.e., using previously trained policies to jumpstart learning in a new task, or by developing curricula [3, 13] of tasks to facilitate the learning of advanced skills or behaviors.”
Reinforcement Learning Control of a Forestry Crane Manipulator (2021, methodology): “This motivates us to deploy a curriculum learning-based [15] solution, in which a sequence of source tasks is carefully combined to form a curriculum of lessons, easing learning of the target task by repeated use of TL.”
NovelGridworlds: A Benchmark Environment for Detecting and Adapting to Novelties in Open Worlds (2021, background): “This functionality provides the capability of training a curriculum for reinforcement learning agents [10].”
Curriculum Learning: A Survey (2021, background): “[35] survey the use of curriculum learning applied to reinforcement learning.”
Adaptive Curriculum Learning (2021, background): “Moreover, curriculum learning is also prevalent in speech recognition [3, 21, 31], natural language [29, 33, 47], and reinforcement learning [10, 13, 22, 25, 26, 39, 40].”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “The difficulty of these curriculums are generally measured difficulty via proxy of agent performance (Narvekar et al., 2020).”
Boosted Curriculum Reinforcement Learning (2022, methodology): “We consider the classic curriculum RL scenario where an agent aims at learning a complex target task by solving a predeﬁned sequence of easier, but increasingly challenging, versions of it (Narvekar et al., 2020).”
A Survey on Curriculum Learning (2021, methodology): “Generating a task curriculum in RL setting often costs greater time than learning the tasks [76].”