Deep learning a grasp function for grasping under gripper pose uncertainty

By Edward Johns, Stefan Leutenegger, Andrew J. Davison. 2016

266 citations recorded by the source corpus.

This paper presents a new method for parallel-jaw grasping of isolated objects from depth images, under large gripper pose uncertainty. Whilst most approaches aim to predict the single best grasp pose from an image, our method first predicts a score for every possible grasp pose, which we denote the grasp function. With this, it is possible to achieve grasping robust to the gripper's pose uncertainty, by smoothing the grasp function with the pose uncertainty function. Therefore, if the single best pose is adjacent to a region of poor grasp quality, that pose will no longer be chosen, and instead a pose will be chosen which is surrounded by a region of high grasp quality. To learn this function, we train a Convolutional Neural Network which takes as input a single depth image of an object, and outputs a score for each grasp pose across the image. Training data for this is generated by use of physics simulation and depth image simulation with 3D object meshes, to enable acquisition of sufficient data without requiring exhaustive real-world experiments. We evaluate with both synthetic and real experiments, and show that the learned grasp score is more robust to gripper pose uncertainty than when this uncertainty is not accounted for.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “el ran 150 times faster than previous methods. Redmon and Angelova (2015) also introduced a grid strategy as YOLO (Redmon et al., 2016) to propose multiple grasp positions in a single input. Further, Johns et al. (2016) used CNN to map the depth image to a grasp function by considering both the unce”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “08, fill in the image using inpainting [24], segment the object using color-based background subtraction, and center the image on the detected object.”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, neutral): “Simulation for robotics has seen success in tasks such as guided control of a camera for object recognition using depth image simulation [16], and grasp pose estimation using physics simulation [17].”
Domain Randomization and Generative Models for Robotic Grasping (2017, methodology): “Others have explored replacing real data with synthetic depth data at training time [14], [23], [24], [47], or combining synthetic RGB images with real images [3].”
Jacquard: A Large Scale Dataset for Robotic Grasp Detection (2018, methodology): “[12], we rendered two more RGB synthetic images with a”
Dex-Net 3.0: Computing Robust Vacuum Suction Grasp Targets in Point Clouds Using a New Analytic Model and Deep Learning (2018, methodology): “To reduce the time-cost of data collection, recent research has proposed to generate labels in simulation using physical models of contact [12], [14].”
Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network (2019, methodology): “[24] used a simulated depth image to predict a grasp outcome for every grasp pose predicted and select the best grasp by smoothing the predicted pose using a grasp uncertainty function.”
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch (2018, methodology): “As an alternative, data-driven approaches have sought to predict grasp outcomes from human supervision [7], [8], simulation [9], [10], [11], or autonomous robotic data collection [2], [3], typically using visual or depth observations.”
Robotic grasp detection using deep convolutional neural networks (2016, methodology): “generated their training data by using a physics simulation and depth image simulation with 3-D object meshes to learn grasp score which is more robust to gripper pose uncertainty [27].”
Learning ambidextrous robot grasping policies (2019, methodology): “Furthermore, the method plans grasps based on depth images, which can be simulated accurately using ray tracing ( 18 , 19 , 23 ) and are invariant to object color ( 24 ).”
Learning Grasping Interaction with Geometry-aware 3D Representations (2017, background): “…grasping planning and control often involves 3D modeling of object shape, modeling dynamics of robot hands, and local surface modeling Goldfeder et al. [2009], Leon et al. [2010], Johns et al. [2016], Varley et al. [2016], Li et al. [2016], Vahrenkamp et al. [2016], Mahler et al. [2016, 2017].”
Deep representation of industrial components using simulated images (2017, methodology): “To increase the volume of training data, we take advantage of a physics-based simulator [12], [13].”
Real-World Multiobject, Multigrasp Detection (2018, background): “Alternatively, [24], [25] predicted grasp scores over the set of all possible grasps (which may be discretized).”
Helping Robots Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality Teleoperation (2020, methodology): “Convolutional neural networks [33] have also been used [34], [32], [35], [36], [37], [38], typically predicting feasibility of a set of grasping hypotheses.”
Learning robust, real-time, reactive robotic grasping (2019, methodology): “…manipulating these objects is an open problem in robotics, we do not consider them for our experiments in order to compare our results with other papers that use similar object classes to ours (Johns et al., 2016; Lenz et al., 2015; Levine et al., 2016; Mahler et al., 2017; Pinto and Gupta, 2016).”
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes? (2017, unknown): “As an alternative, data-driven approaches have sought to predict grasp outcomes from human supervision [15, 16, 7], simulation [17, 18, 19], or autonomous robotic data collection [8, 9], typically using visual or depth observations.”
Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not (2016, methodology): “The complexity of modules in DL model are dramatically increasing in the last several years to be sensitive to minute details so that the abstracted representations are more meaningful and precise for speciﬁc targets (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015).”