Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation

By Suraj Nair, Chelsea Finn. 2019

155 citations recorded by the source corpus.

Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which generates subgoal images conditioned on a goal image, and uses them for planning. The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identifies semantically meaningful states as subgoals. Across three out of four simulated vision-based manipulation tasks, we find that our method achieves nearly a 200% performance improvement over planning without subgoals and model-free RL approaches. Further, our experiments illustrate that our approach extends to real, cluttered visual scenes. Project page: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Others learn to hierarchically solve such long-horizon problems [79, 80, 12, 81, 54].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…of tasks, either by learning sub-goals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016;…”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “Furthermore, we compare to using a goal-image as the task specification instead of language, and provide the agent with a ground truth goal-image of the object in its desired position, with which we use either L2 pixel distance (Pixel) or LPIPS [74] similarity (LPIPS) as a planning cost, reflective ”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, background): “, 2019), or in high-dimensional image space, but from the same scene as the agent (Nair et al., 2018; Nair and Finn, 2019; Singh et al., 2019; Johns, 2021).”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “To mitigate this problem, hierarchical structures have been introduced to neural network methods [8, 22, 24, 25, 27].”
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Recent works have explored learning-based task & motion planning, using methods such as representation learning, hierarchical learning, language as planning space, learning compositional skills and more [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26].”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “Most related is to our method is hierarchical visual foresight (HVF) [48]: while HVF uses MPC to find an action, our approach uses a goal-reaching policy thereby eliminating the cost of running MPC with large dynamics models.”
What can I do here? A Theory of Affordances in Reinforcement Learning (2020, background): “Critical states (Goyal et al., 2019; Nair & Finn, 2019) that are important for decision making would be ideal candidates to include in intent distributions.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “Concurrently, Nair and Finn (2020) design a hierarchical planning method that ﬁnds subgoals via extensive planning.”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “This casts the problem for the high-level policy as finding a sequence of goal states [8, 9, 21, 22].”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, result): “In contrast to prior works that approach this problem from a hierarchical skill learning [7, 8] or planning perspective [9, 10], we show that this “understanding” can be acquired without any explicit predictive model or planning, simply via model-free RL.”