Learning ambidextrous robot grasping policies

By Jeffrey Mahler, Matthew Matl, V. Satish, Michael Danielczuk, Bill DeRose, Stephen McKinley, Ken Goldberg. Science Robotics, 2019

602 citations recorded by the source corpus.

An ambidextrous grasping policy trained on synthetic datasets consistently clears bins of up to 25 novel objects with a physical robot. Universal picking (UP), or reliable robot grasping of a diverse range of novel objects from heaps, is a grand challenge for e-commerce order fulfillment, manufacturing, inspection, and home service robots. Optimizing the rate, reliability, and range of UP is difficult due to inherent uncertainty in sensing, control, and contact physics. This paper explores “ambidextrous” robot grasping, where two or more heterogeneous grippers are used. We present Dexterity Network (Dex-Net) 4.0, a substantial extension to previous versions of Dex-Net that learns policies for a given set of grippers by training on synthetic datasets using domain randomization with analytic models of physics and geometry. We train policies for a parallel-jaw and a vacuum-based suction cup gripper on 5 million synthetic depth images, grasps, and rewards generated from heaps of three-dimensional objects. On a physical robot with two grippers, the Dex-Net 4.0 policy consistently clears bins of up to 25 novel objects with reliability greater than 95% at a rate of more than 300 mean picks per hour.

Subfields: Biomedical Engineering, Control and Systems Engineering, Mechanical Engineering

View Paper: https://www.semanticscholar.org/paper/57c7ced90b8c635f8346983651363379d0357dc2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Control and Systems Engineering

In Mechanical Engineering

Citations in the corpus

Learning Visual Affordances for Robotic Manipulation (2019, background): “0 [69] 312 TossingBot (w/ Placing) 432 TossingBot (w/ Throwing) 514”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, background): “DexNet [19], [20] learns from a large number of depth images of top-down grasps, and gets extremely good performance on grasping novel objects but does not look at longhorizon tasks.”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, methodology): “Robot manipulation using learned, data-driven techniques has become a highly effective paradigm for robot manipulation, as exempliﬁed in the pioneering work of Levine et al. [38], Mahler et al. [42], Florence et al. [20], Kalashnikov et al. [30], and others.”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “0 [21] 312 TossingBot (w/ Placing) 432 TossingBot (w/ Throwing) 514”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “Recent hybrid approaches [27,32] combine analytic models and deep learning tools to synthesize grasps for various end effectors.”
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes (2021, background): “space of possible grasps to planar grasping, where grasps are represented by oriented rectangles around each pixel that define the grasp frame [7, 8, 9].”
MineRL: A Large-Scale Dataset of Minecraft Demonstrations (2019, background): “Similarly, Dex-Net [Mahler et al. , 2019 ] is a dataset of ﬁve million grasps with corresponding 3D pointclouds for robotic manipulation.”
AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains (2022, methodology): “In total, over 300 objects are collected, which is an order of magnitude larger than previous practice [7].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Many works frame grasping as a bandit or classification problem, where the action space consists of discrete grasp candidates and the picking motion is executed open-loop (108, 109).”
ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact (2019, background): “For example, Mahler et al [1], [3], [2], [4] execute those grasps with a robot and record success/failure.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, unknown): “For parallel-jaw grippers, most recent works [47, 61, 66, 79] learn from simulation-based datasets (e.”
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation (2019, methodology): “model-free methods [27], [5], [14], [8] directly evaluate the grasp quality from raw sensor inputs.”