Automatic Goal Generation for Reinforcement Learning Agents

By David Held, Xinyang Geng, Carlos Florensa, Pieter Abbeel. 2017

582 citations recorded by the source corpus.

Reinforcement learning is a powerful technique to train an agent to perform a task. However, an agent that is trained using reinforcement learning is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing. We use a generator network to propose tasks for the agent to try to achieve, specified as goal states. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent. Our method thus automatically produces a curriculum of tasks for the agent to learn. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment. Our method can also learn to achieve tasks with sparse rewards, which traditionally pose significant challenges.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, background): “When learning the forward and reset policies jointly, the training dynamics of our reset policy resemble those of reverse curriculum generation (Florensa et al., 2017), but in reverse.”
Learning quadrupedal locomotion over challenging terrain (2020, methodology): “(39) similarly choose achievable yet difficult goals for RL agents.”
Hindsight Experience Replay (2017, background): “Another approach investigated by Sukhbaatar et al. (2017) and Held et al. (2017) uses self-play between the policy and a task-setter in order to automatically generate goal states which are on the border of what the current policy can achieve.”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, background): “istant start states from a goal. Pinto et al. (2017) use an adversarial framework to perturb the 4 Published as a conference paper at ICLR 2018 environment, inducing improved robustness of the agent. Held et al. (2017) propose a scheme related to our “random Alice” strategy2. 4 EXPERIMENTS The follo”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, neutral): “[16] use a generator network, which is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for an agent, to automatically produce a curriculum of navigation tasks to learn.”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Parameterized rewards are not a new concept, and have been studied previously [34, 19].”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “[5], we postulate that in goal-oriented environments, a strong learning signal is obtained when training on initial states s0 ∼ ρi from which the agent reaches the goal sometimes, but not always.”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, methodology): “ilding have been proposed. For example, the curriculum can be produced by a generator network optimized using adversarial training to produce tasks at the appropriate level of difﬁculty for the agent [35], by applying noise in action space to generate a set of feasible start states increasingly far ”
Hindsight policy gradients (2017, background): “Recent examples of learning goal-conditional behavior include the works of Da Silva et al. [3], Schmidhuber [4], Srivastava et al. [5], Kupcsik et al. [6], Deisenroth et al. [7], Fabisch and Metzen [8], Schaul et al. [9], Zhu et al. [10], Held et al. [11].”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, methodology): “ordering tasks by heuristic measures of difﬁculty can be effective [45], [46]. A line of work has studied automatic discovery of curricula based on learning progress [47], adversarial self-play [18], [48], or backtracking [49]. So far, these methods have not yielded curricula capable of automaticall”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “In this work, we focus on MDPs where only reward dynamics varies [9, 10, 12, 13, 17, 26] and only train the reward decoder (parameterized by θR) as in [56].”
Goal-conditioned Imitation Learning (2019, background): “observation and goal [14]. In theory these approaches could learn how to reach any goal, but the breadth-ﬁrst nature of the algorithm makes that some areas of the space take a long time to be learned [15]. This is specially challenging when there are bottlenecks between different areas of the state-”
Replay-Guided Adversarial Environment Design (2021, methodology): “While our methods do not presuppose a goal-based setting, others have made progress here using generative modeling [12, 29], latent skill learning [14], and exploiting model disagreement [45].”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, unknown): “There are subgoal generation methods that generate intermediate goals such as imagined goal [31, 14] or random goals [34] to help agents solve the tasks.”