Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

By Frederik D. Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex X. Lee, Sergey Levine. 2018

474 citations recorded by the source corpus.

Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Electrical and Electronic Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, background): “…several recent works have incorporated images into real-world (Finn et al., 2016; Finn & Levine, 2017; Babaeizadeh et al., 2017a; Ebert et al., 2017; Piergiovanni et al., 2018; Paxton et al., 2019; Rybkin et al., 2018; Ebert et al., 2018) and simulated (Watter et al., 2015; Hafner et al., 2019) .”
Learning Latent Dynamics for Planning from Pixels (2018, background): “The robotics community focuses on video prediction models for planning (Agrawal et al., 2016; Finn & Levine, 2017; Ebert et al., 2018) that deal with the visual complexity of the real world and solve tasks with a simple gripper, such as grasping or pushing objects.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Nonetheless, prior work has demonstrated good results with model-based methods, particularly in regard to generalization with real-world data (Finn and Levine, 2017; Ebert et al., 2018), and a range of works on predicting physical phenomena have utilized offline datasets (Lerer et al.”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019).”
Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “To evaluate RoboNet’s applicability to different control algorithms, we train a simple version of the inverse model from [11] (refer to Section 3 for details) on a subset of RoboNet containing only Sawyer and Franka data.”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “…2018), are appealing because they allow for algorithms that combine the sample efﬁciency of a model-based approach with the asymptotic performance of high-capacity function approximators, even in domains with high-dimensional observations (Oh et al., 2015; Ebert et al., 2018; Kaiser et al., 2019).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Model-based offline RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “Recently, the success of deep learning has garnered increased interest in learning to solve perception and manipulation tasks in an end-to-end framework [11, 12, 28, 29, 30].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Another direction may be to learn models from observations and specify goals via observations [2], [15], [16], but these may be limited to tasks for which autonomous exploration has a reasonable chance of success.”
Learning Latent Plans from Play (2019, background): “Specifically, it is not to be confused with recent “self-supervised robotics” work, where unattended, scripted robotic data collection is used to learn skills such as grasping ([1, 35, 42, 31, 13, 39]) failure recovery.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Additionally, RL agents often require hand-designed strategies [22, 9] or human demonstrations [38] to overcome hard exploration problems.”
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (2019, background): “For example, in visual-servoing, (SOC1) can be formulated as a goal tracking problem (Ebert et al., 2018), where we control the robot to reach the goal observation x goal , and the objective is to compute a sequence of optimal open-loop actions U that minimizes the cumulative tracking error E […”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “Several works [11, 12, 13, 46] have proposed formulations of visual Model Predictive Control, where the central insight is that a predictive model of sensory input is a powerful signal for learning to perform tasks.”
Control-Aware Representations for Model-based Reinforcement Learning (2020, methodology): “On the other hand, model-based RL algorithms, such as PILCO [Deisenroth and Rasmussen, 2011], MBPO [Janner et al., 2019], and Visual Foresight [Ebert et al., 2018], despite their success, still have many issues in handling the difficulties of learning a model (dynamics) in a high-dimensional…”
Grounding Language in Play (2020, background): “One way of acquiring this kind of control is to ﬁrst learn a model of the environment through interaction (Oh et al., 2015; Hafner et al., 2018; Ebert et al., 2018; Ha and Schmidhuber, 2018) then use it for planning.”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “DVF.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “Byravan et al. (2017); Hafner et al. (2018); Ebert et al. (2018) have shown that visual model predictive control based on such models can be applied to a variety of different settings.”