Domain Randomization and Generative Models for Robotic Grasping

By Joshua Tobin, Wojciech Zaremba, Pieter Abbeel. 2017

191 citations recorded by the source corpus.

Deep learning-based robotic grasping has made significant progress thanks to algorithmic improvements and increased data availability. However, state-of-the-art models are often trained on as few as hundreds or thousands of unique object instances, and as a result generalization can be a challenge. In this work, we explore a novel data generation pipeline for training a deep neural network to perform grasp planning that applies the idea of domain randomization to object synthesis. We generate millions of unique, unrealistic procedurally generated objects, and train a deep neural network to perform grasp planning on these objects. Since the distribution of successful grasps for a given object can be highly multimodal, we propose an autoregressive grasp planning model that maps sensor inputs of a scene to a probability distribution over possible grasps. This model allows us to sample grasps efficiently at test time (or avoid sampling entirely). We evaluate our model architecture and data generation pipeline in simulation and the real world. We find we can achieve a >90% success rate on previously unseen realistic objects at test time in simulation despite having only been trained on random objects. We also demonstrate an 80% success rate on real-world grasp attempts despite having only been trained on random simulated objects.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, methodology): “Domain randomization was also used to plan robust grasps [65, 66, 107] and to transfer learned locomotion [105] and grasping [123] policies for relatively simple robots.”
Learning dexterous in-hand manipulation (2018, methodology): “Domain randomization was also used to plan robust grasps [38, 39, 63] and to transfer learned locomotion [61] and grasping [69] policies for relatively simple robots.”
Learning task-oriented grasping for tool manipulation from simulated self-supervision (2018, methodology): “bjects suitable for tool manipulation while exhibiting rich variations in terms of their geometric and physical properties. As shown in Figure4, we leverage a common strategy of procedural generation [3, 49] to produce a large set of diverse and realistic objects that can be used as tools for tasks ”
Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network (2019, background): “he objects detected in the camera’s ﬁeld of view. A. Overview We present a unique and generalized solution for the problem of robotic grasping. Unlike the previous work done in robotic grasping [23], [34] where the required grasp is predicted as a grasp rectangle calculated by choosing the best gras”
Autonomous navigation of stratospheric balloons using reinforcement learning (2020, methodology): “We address the problem of partial observability 9,38 by incorporating uncertainty variables as additional inputs to the neural network, and used a relative encoding of the wind column to reduce the gap between simulation and reality 6,7 .”
Sim2Real Viewpoint Invariant Visual Servoing by Recurrent Control (2018, methodology): “The randomization procedure described in the previous section already provides some degree of generalization [33, 30].”
Learning to Manipulate Deformable Objects without Demonstrations (2019, methodology): “To transfer our policies to the real-robot, we use domain randomization (DR) [62, 45, 48] in the simulator along with using images of real cloths.”
Sim2Real View Invariant Visual Servoing by Recurrent Control (2017, methodology): “In fact, prior methods observed that simply randomizing the environment can enable manipulation of simple geometric shapes in the real world [33].”
Synthetic Data for Deep Learning (2019, background): “[587] consider robotic grasping, a problem where the lack of real data is especially dire (see also Sections 3.”