Visual Reinforcement Learning with Imagined Goals

By Ashvin Nair, Vitchyr H. Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, Sergey Levine. 2018

603 citations recorded by the source corpus.

For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised "practice" phase where it imagines goals and attempts to achieve them. We learn a visual representation with three distinct purposes: sampling goals for self-supervised practice, providing a structured transformation of raw sensory inputs, and computing a reward signal for goal reaching. We also propose a retroactive goal relabeling scheme to further improve the sample-efficiency of our method. Our off-policy algorithm is efficient enough to learn policies that operate on raw image observations and goals for a real-world robotic system, and substantially outperforms prior techniques.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “While prior work (Steenbrugge et al., 2018; LaversanneFinot et al., 2018; Nair et al., 2018; Higgins et al., 2017b, 2018b) successfully applied disentanglement methods such as β-VAE on a variety of downstream tasks, it is not clear to us that these approaches and trained models performed well…”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Our trajectory-collecting agent uses both random walks and a simultaneously learned curiosity-driven goalconditioned policy (Ghosh et al., 2018; Nair et al., 2018).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “sequence modeling, curiosity-based exploration or even domain adaptation in reinforcement learning [45, 25, 37, 16, 18, 29, 47].”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “While prior work (Steenbrugge et al., 2018; Laversanne-Finot et al., 2018; Nair et al., 2018; Higgins et al., 2017b; 2018) successfully applied disentanglement methods such as β-VAE on a variety of downstream tasks, it is not clear to us that these approaches and trained models performed well…”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Our work builds on goal-conditioned RL [17, 18, 19, 20], which has been explored in the context of reward-free learning [21], learning with sparse rewards [18], large scale generalizable imitation learning [22], and hierarchical RL [15].”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, neutral): “We refer to [44, 45, 46, 47] for recent works towards this challenge.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “To make the interface simple and accessible, we base our suite on the Multiworld interface [52] and the OpenAI Gym environment interfaces [11], making additions and adaptations of the suite relatively easy for researchers already familiar with Gym.”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “required to solve subproblems seen in the past in the context of different problems. For example, we can view latent variable reinforcement learning architectures such as (Ha & Schmidhuber, 2018; Nair et al., 2018) as simultaneously solving an image reconstruction problem and an action predictio”
Disentangling Factors of Variation Using Few Labels (2019, background): “[46] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, methodology): “While one can use HER with generative models of images [48, 45, 38], the resulting representation of the state may not effectively capture the relevant aspects of the desired task.”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “When the goal is a target observation, most methods acquire new goals by sampling observations previously generated in the environment: Nair et al. (2018) generate visual goals by training a Variational Auto-Encoder (Kingma and Welling, 2014) over the generated experience.”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, background): “…have demonstrated that models with disentangled representations show improvements in terms of these shortcomings (Higgins et al., 2017b; 2018b; Achille et al., 2018; Steenbrugge et al., 2018; Nair et al., 2018; Laversanne-Finot et al., 2018; van Steenkiste et al., 2019; Locatello et al., 2019).”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “Previous efforts have explored image-based goal specification, with significant successes in visual navigation and manipulation tasks (Pathak et al., 2018; Nair et al., 2018; Fu et al., 2018; Singh et al., 2019).”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Goal images are frequently used to specify rewards in RL [3, 33, 37, 70, 77, 78, 86, 114, 131].”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “This casts the problem for the high-level policy as finding a sequence of goal states [8, 9, 21, 22].”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Explicitly addressing partial observability is outside the scope of our work, so we follow the convention in prior work by assuming full observability and refer to images as states (Lillicrap et al., 2016; Nair et al., 2018).”