Self-Supervised Visual Planning with Temporal Skip Connections

By Frederik D. Ebert, Chelsea Finn, Alex X. Lee, Sergey Levine. 2017

362 citations recorded by the source corpus.

In order to autonomously learn wide repertoires of complex skills, robots must be able to learn from their own autonomously collected data, without human supervision. One learning signal that is always available for autonomously collected data is prediction: if a robot can learn to predict the future, it can use this predictive model to take actions to produce desired outcomes, such as moving an object to a particular location. However, in complex open-world scenarios, designing a representation for prediction is difficult. In this work, we instead aim to enable self-supervised robotic learning through direct video prediction: instead of attempting to design a good representation, we directly predict what the robot will see next, and then use this model to achieve desired goals. A key challenge in video prediction for robotic manipulation is handling complex spatial arrangements such as occlusions. To that end, we introduce a video prediction model that can keep track of objects through occlusion by incorporating temporal skip-connections. Together with a novel planning criterion and action space formulation, we demonstrate that this model substantially outperforms prior work on video prediction-based control. Our results show manipulation of objects not seen during training, handling multiple objects, and pushing objects around obstructions. These results represent a significant advance in the range and complexity of skills that can be performed entirely with self-supervised robotic learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “Finally, we tried sampling temperature 0.9 and 1.0 only on the BAIR Robot Pushing validation set and found that temperature 0.9 consistently gave more robust predictions and better results on all extrinsic metrics.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “Plenty of methods have been proposed for video prediction on the BAIR Robot dataset (Finn et al., 2016; Ebert et al., 2017; Babaeizadeh et al., 2017; Denton et al., 2017; Denton & Fergus, 2018; Lee et al., 2018) where the future frames are predicted given the past frame(s) and (or) action(s) of a…”
Model-Based Reinforcement Learning for Atari (2019, background): “Though most of such works do not use image observations, several recent works have incorporated images into real-world (Finn et al., 2016; Finn & Levine, 2016; Babaeizadeh et al., 2017; Ebert et al., 2017; Piergiovanni et al., 2018; Paxton et al., 2018; Rybkin et al., 2018; Ebert et al., 2018) and s”
Learning Latent Dynamics for Planning from Pixels (2018, background): “The robotics community focuses on video prediction models for planning (Agrawal et al., 2016; Finn and Levine, 2017; Ebert et al., 2017) that deal with the visual complexity of the real world and solve tasks with simple underlying dynamics, such as grasping or pushing objects.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “…et al., 2016; Finn et al., 2016; Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016), or from a symbolic, abstract…”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “In addition, the ﬁrst frame of the sequence is also given as one of the intermediate frames [7].”
Stochastic Variational Video Prediction (2017, methodology): “We quantitatively and qualitatively evaluate SV2P on following real-world datasets: • BAIR robot pushing dataset (Ebert et al., 2017): This dataset contains action-conditioned videos collected by a Sawyer robotic arm pushing a variety of objects.”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Model-based methods that predict images [48, 10, 8, 28] or learn inverse models [1] can also accommodate various goals, but tend to limit the horizon length due to model drift.”
Stochastic Video Generation with a Learned Prior (2018, methodology): “We evaluate our SVG-FP and SVG-LP model on one synthetic video dataset (Stochastic Moving MNIST) and two real ones (KTH actions (Schuldt et al., 2004) and BAIR robot (Ebert et al., 2017)).”
Learning Latent Plans from Play (2019, background): “Learning powerful goalconditioned policies with reinforcement learning can produce policies with good long-horizon performance, but is difficult in terms of both the number of samples required and the need for extensive on-policy exploration [43, 42, 31, 12, 14, 29, 33, 30].”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, background): “Beyond selection of examples from a pre-determined set, recent work in robotics [2, 7, 10, 36] study learning tasks with interactive visuo-motor setups such as robotic arms.”
Stochastic Adversarial Video Prediction (2018, methodology): “The generator is a convolutional LSTM [27,47] that predicts pixel-space transformations between the current and next frame, with additional skip connections with the ﬁrst frame as done in SNA [4].”
Zero-Shot Visual Imitation (2018, background): “Forward/Inverse Dynamics and Consistency Numerous prior works, such as Ebert et al. (2017); Oh et al. (2015); Watter et al. (2015), have learned forward dynamics model for planning actions.”
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning (2018, methodology): “In contrast to prior self-supervised visual planning works [4, 5], we can perform substantially longer tasks, by using image registration with respect to a goal image.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (Ebert et al., 2017; Chua et al., 2018; Parmas et al., 2019), rather than leveraging analytic gradients offered by neural network dynamics (Henaff et al., 2019; Srinivas et al., 2018).”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, methodology): “on a standard benchmark for video prediction, the BAIR robot pushing dataset [36], in addition to our DIGIT tactile marble manipulation videos.”
Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning (2020, methodology): “Many model-based RL approaches [28, 4, 21, 10, 6] use a random-sampling based planner (e.g. cross-entropy method) to solve the underlying MPC problem.”