Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
By Sainbayar Sukhbaatar, Emily L. Denton, Arthur Szlam, Rob Fergus. 2018
47 citations recorded by the source corpus.
In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play from Sukhbaatar et al. (2018), that automatically learns a good representation of sub-goals in the environment and a low-level policy that can execute them. A high-level policy can then direct the lower one by generating a sequence of continuous sub-goal vectors. We evaluate our model using Mazebase and Mujoco environments, including the challenging AntGather task. Visualizations of the sub-goal embeddings reveal a logical decomposition of tasks within the environment. Quantitatively, our approach obtains compelling performance gains over non-hierarchical approaches.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Caiming Xiong
- Richard Socher
- Shimon Whiteson
- Chongjie Zhang
- Tonghan Wang
- Mohit Bansal
- Bei Peng
- Luke S. Zettlemoyer
Citations in the corpus
- RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reus”
- Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards (2019, background): “Similar benefits have been gained through asymmetric self-play with goal-directed tasks (Sukhbaatar et al., 2018a,b).”
- Learning to Prove Theorems by Learning to Generate Theorems (2020, result): “Automatic goal generation by self-play Our work is similar to the line of work in reinforcement learning (Florensa et al., 2018; Sukhbaatar et al., 2017; 2018; Durugkar & Stone, 2018) that deploys one agent to generate tasks for another agent to accomplish.”
- Reinforcement Learning for Combinatorial Optimization: A Survey (2020, background): “The representation of the primary goals and primitive subgoals can also vary, from [Sukhbaatar et al. , 2018, Rafati and Noelle, 2019] to the handcrafted ones [Bacon et al. , 2017], such as the particular intermediate states of the environment, for example.”