Visual Robot Task Planning

By Chris Paxton, Yotam Barnoy, Kapil D. Katyal, Raman Arora, Gregory Hager. 2018

58 citations recorded by the source corpus.

Prospection is key to solving challenging problems in new environments, but it has not been deeply explored as applied to task planning for perception-driven robotics. We propose visual robot task planning, where we take in an input image and must generate a sequence of high-level actions and associated observations that achieve some task. In this paper, we describe a neural network architecture and associated planning algorithm that (1) learns a representation of the world that can generate prospective futures, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) evaluates these actions via a variant of Monte Carlo Tree Search to find a viable solution to a particular problem. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information, and used this to generate and visualize task plans on held-out examples of a block-stacking simulation.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, methodology): “…several recent works have incorporated images into real-world (Finn et al., 2016; Finn & Levine, 2017; Babaeizadeh et al., 2017a; Ebert et al., 2017; Piergiovanni et al., 2018; Paxton et al., 2019; Rybkin et al., 2018; Ebert et al., 2018) and simulated (Watter et al., 2015; Hafner et al., 2019) .”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “In particular, the ability to visually hallucinate future frames has enabled applications in robotics [8] and healthcare [26].”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, background): “Visual MPC, as well as other video prediction-based planners [9, 16, 35, 45], generally do not succeed at such temporally extended tasks.”
Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image (2020, methodology): “For example, in [35] the considered scene contains always the same four objects with the same colors, which allows them to have a fixed input vector of separate actions for all objects.”
Prospection: Interpretable plans from language by predicting the future (2019, methodology): “In addition, our approach is compatible with work in Visual Robot Task Planning [4], which shows that prospective subgoal predictions can be used to generate task plans.”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “…of video prediction frameworks have been developed speciﬁcally for robotics (Babaeizadeh et al., 2017; Lee et al., 2018; Ebert et al., 2017), which combined with planning have been used to complete diverse behaviors (Nair et al., 2018b; Ebert et al., 2018b; Paxton et al., 2018; Xie et al., 2019).”
Time Reversal as Self-Supervision (2018, background): “These models have been shown to be effective in planning [29], and have even been extended to operate in 3D point cloud space [30], [31].”