Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning

By Frederik D. Ebert, Sudeep Dasari, Alex X. Lee, Sergey Levine, Chelsea Finn. 2018

67 citations recorded by the source corpus.

Prediction is an appealing objective for self-supervised learning of behavioral skills, particularly for autonomous robots. However, effectively utilizing predictive models for control, especially with raw image inputs, poses a number of major challenges. How should the predictions be used? What happens when they are inaccurate? In this paper, we tackle these questions by proposing a method for learning robotic skills from raw image observations, using only autonomously collected experience. We show that even an imperfect model can complete complex tasks if it can continuously retry, but this requires the model to not lose track of the objective (e.g., the object of interest). To enable a robot to continuously retry a task, we devise a self-supervised algorithm for learning image registration, which can keep track of objects of interest for the duration of the trial. We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation. Our real-world experiments demonstrate that a model trained with 160 robot hours of autonomously collected, unlabeled data is able to successfully perform complex manipulation tasks with a wide range of objects not seen during training.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Self-supervised techniques have a variety of applications in a broad range of computer vision topics [14, 35, 7, 29, 34].”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “com/google/revisiting-self-supervised In robotics, both the result of interacting with the world, and the fact that multiple perception modalities simultaneously get sensory inputs are strong signals which can be exploited to create self-supervised tasks [21, 41, 26, 10].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “We most commonly use a diagonal Gaussian combined the automatic grasping primitive developed in [48].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “This article combines and extends material from several prior conference papers [6], [7], [8], [9], presenting them in the context of a unified system.”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, methodology): “Once the subgoals are found, then the agent simply plans using visual MPC (Finn & Levine, 2016; Ebert et al., 2018b) from each s k − 1 to s k until a cost threshold is reached or for a ﬁxed, maximum number of timesteps, then from s ∗ k to s k +1 , until planning to the goal state, where s ∗ k is…”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, background): “This explains why models of this kind have excelled in physically and visually restricted environments like those used in robotics [20], but dramatically underfit on somewhat more diverse data [17, 39].”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, methodology): “We evaluate OP3 on the dataset from Ebert et al. [60] which contains videos of a robotic arm moving cloths and other deformable and multipart objects with varying textures.”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, methodology): “To remedy this, recent work [12] has designed better planning cost functions to allow the robot to continuously retry a task.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, methodology): “…model-based approaches [26, 27], which explicitly learn the environment forward dynamics from images have also been employed for multi-task vision-based robotic manipulation tasks, either with planning algorithms [28, 29, 30, 31, 32, 33, 34, 35] or for optimizing a parametric policy [36].”
The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, unknown): “BAIR VisInt-solid/cloth[32] 3 7 7 – 3 7 7 7 3 3 3 3 3 16k/31k 30/20 –”
Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects (2019, background): “However, a series of efforts [24], [25] has been made to find informative losses on images, which are required for long-horizon planning in the model predictive control framework.”