Reverse Curriculum Generation for Reinforcement Learning

By Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, Pieter Abbeel. 2017

535 citations recorded by the source corpus.

Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration. For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock. These goal-oriented tasks present a considerable challenge for reinforcement learning, since their natural reward function is sparse and prohibitive amounts of exploration are required to reach the goal and receive some learning signal. Past approaches tackle these problems by exploiting expert demonstrations or by manually designing a task-specific reward shaping function to guide the learning agent. Instead, we propose a method to learn these tasks without requiring any prior knowledge other than obtaining a single state in which the task is achieved. The robot is trained in reverse, gradually learning to reach the goal from a set of start states increasingly far from the goal. Our method automatically generates a curriculum of start states that adapts to the agent's performance, leading to efficient training on goal-oriented tasks. We demonstrate our approach on difficult simulated navigation and fine-grained manipulation problems, not solvable by state-of-the-art reinforcement learning methods.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Curriculum Learning (Bengio et al., 2009; Florensa et al., 2017; Zhang et al., 2017b): Presents the learning agent with progressively more complex task settings, such that it can grasp gradually more sophasticated skills.”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “When learning the forward and reset policies jointly, the training dynamics of our reset policy resemble those of reverse curriculum generation (Florensa et al., 2017), but in reverse.”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, background): “If demonstrations are not available but the environment can be reset to arbitrary states, one way to learn goal-reaching but avoid using demonstrations is to reuse successful rollouts as in [40].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[27] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “The list of potential improvements is long e.g. Prioritized Experience Replay (Schaul et al., 2015b), distributional RL (Bellemare et al., 2017), entropy-regularized RL (Schulman et al., 2017a), or reverse curriculum generation (Florensa et al., 2017).”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, background): “ by requiring the environment to be either reversible or resettable. Several concurrent works are related: Andrychowicz et al. (2017) form an implicit curriculum by using internal states as a target. Florensa et al. (2017) automatically generate a series of increasingly distant start states from a g”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, methodology): “Related approaches were recently experimented in the context of Deep Reinforcement Learning, such as in Hindsight Experience Replay [21] and Reverse Curriculum Learning [22] (however using monolithic goal parameterized policies), and within the Power Play framework [23].”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, methodology): “tasks at the appropriate level of difﬁculty for the agent [35], by applying noise in action space to generate a set of feasible start states increasingly far from the goal (i.e. a reverse curriculum) [36], or through intrinsically motivated goal exploration processes (IMGEPs) [16]. The POWERPLAY alg”
Hindsight policy gradients (2017, background): “Consequently, developing goal-based curricula to facilitate learning has also attracted considerable interest (Fabisch & Metzen, 2014; Florensa et al., 2017; Sukhbaatar et al., 2018; Srivastava et al., 2013).”
Automated curriculum generation for Policy Gradients from Demonstrations (2019, methodology): “This category ranges from simple methods like behavioral cloning to more complex ones like DAGGER Ross et al. [2010] that have a human involved in the process.”
Human-in-the-loop machine learning: a state of the art (2022, methodology): “Reinforcement Learning (RL) Teacher adopt reinforcement learning models as the teacher to play dynamic data selection according to the feedback from the student.”
Multi-task learning for natural language processing in the 2020s: where are we going? (2020, background): “Curriculum learning has not been explored much in NLP or MTL, however, it has a rich history in reinforcement learning (RL) where curriculum is used to guide trained agents to more complex and realistic behaviors (Svetlik et al., 2017; Florensa et al., 2017).”
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018, background): “A line of work [9, 5, 4, 7, 3] takes this approach in the context of intrinsic motivation.”
Automatic Goal Generation for Reinforcement Learning Agents (2017, background): “More recent work overcomes some of these limitations by training low-level modulated locomotor controllers (Heess et al., 2016), or multimodal policies with an information theoretic regularizer to learn a ﬁxed-size set of skills (Florensa et al., 2017).”
Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning (2020, unknown): “complex task instances to facilitate learning [29][30][31].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Such methods often utilize curricula provided by human designers [28, 322, 300], though several recent methods seek to automatically generate a curriculum [273, 374, 115].”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “…be explicitly generated by: searching the space (Schmidhuber, 2013), teacher-student setting (Matiisen et al., 2019), increasing distance between the starting point and goal (Jabri et al., 2019) or using a density model to generate a task distribution for the meta learner (Florensa et al., 2017).”
POET: open-ended coevolution of environments and their optimized solutions (2019, background): “across a variety of related fields, such as goal generation [15] and reverse curriculum generation in RL [16], intrinsically motivated goal exploration processes (IMGEPs) [17], the POWERPLAY search for an increasingly-general problem solver through new tasks [49], and Teacher-Student Curriculum Lear”