Deep learning for detecting robotic grasps

By Ian Lenz, Honglak Lee, Ashutosh Saxena. 2013

1,772 citations recorded by the source corpus.

We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast and robust, we present a two-step cascaded system with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Second, we need to handle multimodal inputs effectively, for which we present a method that applies structured regularization on the weights based on multimodal group regularization. We show that our method improves performance on an RGBD robotic grasping dataset, and can be used to successfully execute grasps on two different robotic platforms.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, background): “The high dimensionality of the network also makes reinforcement learning very difficult [3].”
Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, neutral): “Our work complements previous works on grasping items [30, 21], pick and place tasks [11], and detecting bar code for grocery checkout [16].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Engineered success detectors can operate on ground-truth state in simulation, while learned success detectors can be trained on real examples of successes and failures in the real world [84, 85, 86, 87, 88].”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, background): “Deep neural networks have achieved outstanding results in various applications including computer vision [13, 29, 31, 32, 38], natural language processing [45, 47], speech recognition [5, 10], robotics [25], bioinformatics [6], and many others.”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “Real-world robotic manipulation requires broad generalization, and indeed much of the research on robotic grasping has sought to achieve such generalization, either through the use of grasp metrics based on first principles [20] or learning [21, 10], with the latter class of methods achieving some o”
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection (2016, methodology): “Data-driven methods take a variety of different forms, including human-supervised methods that predict grasp configurations (Herzog et al., 2014; Lenz et al., 2015) and methods that predict finger placement from geometric criteria computed offline (Goldfeder et al., 2009a).”
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours (2015, background): “On the other hand, [23], [24] use human annotated grasp data to train grasp synthesis models over RGB-D data.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “There is a long history of robotics benchmarks [28], datasets [29, 30, 31, 32, 33, 34, 35], competitions [36] and standardized object sets [37, 38] that have played an important role in robotics research.”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, background): “In the realm of learning from manipulator data, Lenz et al. [18] presented one of the ﬁrst attempts to collect grasping data using expert annotations.”
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018, background): “Instead of relying exclusively on an analytic understanding of the physics of an object, data-driven methods seek to directly predict either human-specified grasp positions [8] or empirically estimated grasp outcomes [5], [6].”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “also trained on the Cornell Grasping Dataset [31] (containing 8,019 examples) and achieved a 93% recognition rate using grayscale images and an 80−20 imagewise training-validation”
Supervision via competition: Robot adversaries for learning tasks (2016, methodology): “Amazingly, it has been shown that given a specific task, if enough amount of data is collected, these frameworks can outperform manually designed mathematical models [1]–[4].”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, methodology): “Recent papers have focused on data-driven approaches that directly learn a mapping from visual observations to grasp control [25, 4, 7].”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, methodology): “Due to this limitation, data-driven methods have become the dominant way to solve grasping [33, 37].”
Improving object detection with deep convolutional networks via Bayesian optimization and structured prediction (2015, background): “, car, pedestrian, bicycles) for autonomous driving [17], detecting objects for robotic grasping or manipulation in robotic surgery or manufacturing [28], and many others.”
Robot Learning Manipulation Action Plans by "Watching" Unconstrained Videos from the World Wide Web (2015, background): “Recently, several learning based systems were reported that infer contact points or how to grasp an object from its appearance (Saxena, Driemeyer, and Ng 2008; Lenz, Lee, and Saxena 2014).”