Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
By Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg. 2017
1,418 citations recorded by the source corpus.
To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net .
Subfields: Biomedical Engineering, Control and Systems Engineering, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biomedical Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Andy Zeng
- Dieter Fox
- Abhinav Kumar Gupta
- Silvio Savarese
- Kristen Grauman
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Dieter Fox
- Andy Zeng
- Silvio Savarese
- Li Fei-Fei
- Abhinav Kumar Gupta
In Robotics & Control
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Silvio Savarese
- Dieter Fox
- Abhinav Kumar Gupta
- Andy Zeng
- Li Fei-Fei
Citations in the corpus
- Solving Rubik's Cube with a Robot Hand (2019, background): “d to dynamics randomization [80, 3, 105, 119, 77] to learn a robust policy that transfers to similar environments but with different dynamics. Domain randomization was also used to plan robust grasps [65, 66, 107] and to transfer learned locomotion [105] and grasping [123] policies for relatively si”
- Driving Policy Transfer via Modularity and Abstraction (2018, background): “to facilitate transfer: for example, variants of domain adaptation [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color images is known to simplify transfer [50,28,27]. When dealing with color images, domain randomization [46,21,42,43] enables direct sim-to”
- Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, background): “rience [7], [8] or induced by grasp stability metrics [9], [10].”
- IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, methodology): “Large-Scale Data Collection in Robotics : Self-Supervised Learning has been employed to collect and learn from large amounts of data for tasks such as grasping in both simulated [13, 18, 25] and physical settings [17, 22, 29].”
- ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “SSL has been used successfully to collect and learn from large quantities of data in both simulated [7, 8, 9] and physical settings [5, 10, 11] for tasks such as grasping.”
- Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018, background): “A number of methods combine both ideas, for example using analytic metrics to label training data [3], [7].”
- TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, neutral): “0 [16] 250 FC-GQ-CNN [22] 296 Dex-Net 4.”
- Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, methodology): “0 from Mahler et al. [24] ( DexNet ) for a strong real-world grasping baseline.”
- Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, methodology): “State-of-the-art grasping systems typically either operate in an open-loop style, where grasping locations are chosen, and then a motion is executed to complete the grasp [69, 41, 37, 60], or in a closed-loop manner, where grasp prediction is continuously run during motion, either explicitly [65], o”
- BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills and objects [42, 43, 44, 45].”
- Prospection: Interpretable plans from language by predicting the future (2019, background): “23], [24] which harness the unique perceptual and manipulation capabilities of robotics. While accurate grasping and placement is not a focus of our work, it has been explored in the literature [25], [26], [27]. In particular, high-precision grasping with deep neural networks generally takes the for”
- Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods (2018, background): “Deep learning has been recently incorporated into such systems [20], [22], [24], [26], [32].”
- Learning ambidextrous robot grasping policies (2019, methodology): “The policy optimizes for the highest-qualitygraspforeachgripperseparately,usingthecross-entropymethod (CEM)( 10 , 19 , 27 , 56 ), and then selects the grasps with the highest estimated quality across the grippers.”
- Dex-Net 3.0: Computing Robust Vacuum Suction Grasp Targets in Point Clouds Using a New Analytic Model and Deep Learning (2018, methodology): “[20] demonstrated that a GQ-CNN trained on Dex-Net 2.”
- A System for General In-Hand Object Re-Orientation (2021, background): “makes object grasping an easy task, while many prior works [28, 29] require much more involved training techniques to learn grasping skills with parallel-jaw grippers.”
- Real-World Multiobject, Multigrasp Detection (2018, background): “Alternatively, [24], [25] predicted grasp scores over the set of all possible grasps (which may be discretized).”