Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning

By Lucas Manuelli, Yunzhu Li, Peter R. Florence, Russ Tedrake. 2020

113 citations recorded by the source corpus.

Predictive models have been at the core of many robotic systems, from quadrotors to walking robots. However, it has been challenging to develop and apply such models to practical robotic manipulation due to high-dimensional sensory observations such as images. Previous approaches to learning models in the context of robotic manipulation have either learned whole image dynamics or used autoencoders to learn dynamics in a low-dimensional latent state. In this work, we introduce model-based prediction with self-supervised visual correspondence learning, and show that not only is this indeed possible, but demonstrate that these types of predictive models show compelling performance improvements over alternative methods for vision-based RL with autoencoder-type vision training. Through simulation experiments, we demonstrate that our models provide better generalization precision, particularly in 3D scenes, scenes involving occlusion, and in category-generalization. Additionally, we validate that our method effectively transfers to the real world through hardware experiments. Videos and supplementary materials available at this https URL

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Li Fei-Fei
  2. Andrew Zisserman
  3. Silvio Savarese
  4. Trevor Darrell
  5. Abhinav Kumar Gupta
  6. Luc Van Gool
  7. Jitendra Malik
  8. Cordelia Schmid

In Control and Systems Engineering

  1. Li Fei-Fei
  2. Silvio Savarese
  3. Ajay Mandlekar
  4. Danfei Xu
  5. Andrew Zisserman
  6. Ankush Gupta
  7. Rui Wang
  8. Chen Wang

Citations in the corpus