Deep visual foresight for planning robot motion

By Chelsea Finn, Sergey Levine. 2016

888 citations recorded by the source corpus.

A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation — pushing objects — and can handle novel objects not seen during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “The robotics community focuses on video prediction models for planning (Agrawal et al., 2016; Finn and Levine, 2017; Ebert et al., 2017) that deal with the visual complexity of the real world and solve tasks with simple underlying dynamics, such as grasping or pushing objects.”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “Finn & Levine (2017) train an action-conditional video-prediction network and use it for model-predictive control (MPC) of a robot arm.”
Value Prediction Network (2017, methodology): “For continuous control problems, deep learning has been combined with model predictive control (MPC) [6, 19], a specific way of using an observation-prediction model.”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “e model predicts are likely to yield a desired state of the world. For a recent example of such a model, see Finn et al. (2016b), and for a recent example of the use of such a model for planning, see Finn and Levine (2016). Another way that generative models might be used for reinforcement learning ”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Nonetheless, prior work has demonstrated good results with model-based methods, particularly in regard to generalization with real-world data (Finn and Levine, 2017; Ebert et al., 2018), and a range of works on predicting physical phenomena have utilized offline datasets (Lerer et al., 2016;…”
MagNet: A Two-Pronged Defense against Adversarial Examples (2017, background): “They are used in autonomous control for robots and vehicles [1, 3, 4], financial systems [32], medical treatments [31], information security [12, 29], and human-computer interaction [11, 13].”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…fact that conditioning on high-dimensional data such as images would require modeling and generating images in order to plan over several timesteps (Finn & Levine, 2017), and to the fact that modeling images is challenging and may un-necessarily focus on visual details which are not relevant for…”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, neutral): “Several works have proposed to use deep neural network models for building predictive models of images [39, 8], but these methods have either required extremely large datasets for training [39] or were applied to short-horizon control tasks [8, 38].”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “We demonstrate the utility of our data by building on the visual foresight approach [9, 7], as it further enables generalization across tasks without requiring reward signals.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “This article combines and extends material from several prior conference papers [6], [7], [8], [9], presenting them in the context of a uniﬁed system.”
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, neutral): “based [1], [2], [3] or data-driven [4], [5], [6] approaches.”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Model-based offline RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling.”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, methodology): “A common approach is to learn a neural model of the environment, including from raw observations, and use it in classical planning algorithms such as trajectory optimization [26–28].”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “…required a combination with model-free techniques (Oh et al., 2015; Heess et al., 2015; Nagabandi et al., 2017), or domain-speciﬁc policy learning or planning algorithms (Lenz et al., 2015; Agrawal et al., 2016; Pinto & Gupta, 2016; Levine et al., 2016b; Finn & Levine, 2017; Nair et al., 2017).”
Stochastic Variational Video Prediction (2017, background): “Furthermore, such models can be inherently useful themselves, for example, to allow an autonomous agent or robot to decide how to interact with the world to bring about a desired outcome (Oh et al., 2015; Finn & Levine, 2017).”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Model-based methods that predict images [48, 10, 8, 28] or learn inverse models [1] can also accommodate various goals, but tend to limit the horizon length due to model drift.”
Learning Latent Plans from Play (2019, background): “Learning powerful goalconditioned policies with reinforcement learning can produce policies with good long-horizon performance, but is difficult in terms of both the number of samples required and the need for extensive on-policy exploration [43, 42, 31, 12, 14, 29, 33, 30].”