Real-time grasp detection using convolutional neural networks

By Joseph Redmon, Anelia Angelova. 2014

874 citations recorded by the source corpus.

We present an accurate, real-time approach to robotic grasp detection based on convolutional neural networks. Our network performs single-stage regression to graspable bounding boxes without using standard sliding window or region proposal techniques. The model outperforms state-of-the-art approaches by 14 percentage points and runs at 13 frames per second on a GPU. Our network can simultaneously perform classification so that in a single step it recognizes the object and finds a good grasp rectangle. A modification to this model predicts multiple grasps per object by using a locally constrained prediction mechanism. The locally constrained model performs significantly better, especially on objects that can be grasped in a variety of ways.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

In Robotics & Control

Citations in the corpus

You Only Look Once: Unified, Real-Time Object Detection (2015, neutral): “Our work is similar in design to work on grasp detection by Redmon et al [26].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “ PREDICTION PRECISION. Approach Sensor Deep Dataset Real-time Effect Application Training Precision Model Source Test Efﬁciency Lenz et al. (2015b) RGBD DNN Benchmark X Object Detection Grasping FF F Redmon and Angelova (2015) RGBD CNN Benchmark X Object Detection Grasping F FF Li et al. (2016) 3D L”
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, methodology): “A common approach to deploying these methods in real systems involves pre-computing grasps from a database of known 3D object models [23], and indexing them at run-time with point cloud registration for object pose estimation [24], [25].”
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection (2016, background): “Both types of data-driven grasp selection have recently incorporated deep learning (Kappler et al., 2015; Lenz et al., 2015; Redmon & Angelova, 2015).”
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours (2015, background): “Hence the grasp conﬁguration lies in 3 dimensions, ( x, y ) : position of grasp point on the surface of table and θ : angle of grasp.”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “[31] created a dataset of over 1k RGB-D images with human labels of successful and unsuccessful grasping regions, which has been used to train fast CNN-based detection models [49].”
A deep-network solution towards model-less obstacle avoidance (2016, background): “Since the last decade, deep-network structures have been adopted ubiquitously not only in robotics [1], [2], [3], [4], but also natural language processing [5], [6], [7], [8], computer vision [9] and so on.”
Deep learning a grasp function for grasping under gripper pose uncertainty (2016, methodology): “We follow the trend in this setup [6], [8], [12], [22] by constraining the gripper to a perpendicular orientation with respect to a ﬂat table surface upon which the object is resting.”
AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains (2022, background): “…for grasp pose detection evolved with the development of learning-based methods, including a point-based representation [15], [16], [17], a point pair representation [18], a rectangle representation [5], [19], [20], [21], [22], [23], [24], a grasp quality maps representation [25], etc.”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “Redmon and Angelova (2015) proposed a larger neural network, which performs a single-stage regression to obtain graspable bounding boxes without using standard sliding window or region proposal techniques.”
Grasp Pose Detection in Point Clouds (2017, methodology): “he gripper must approach the grasp target from a direction roughly orthogonal to the image. Several other approaches fall into this category as well including (Lenz et al. 2015; Pinto and Gupta 2015; Redmon and Angelova 2015; Fischinger and Vincze 2012). In particular, Redmon and Angelova use the sa”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, methodology): “Visual affordances A few methods infer visual affordances for grasping with simple grippers [40], [19], [23], [20] and explore non-robotics affordances [32], [9], [34], [11].”
Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network (2019, methodology): “Unlike the previous work done in robotic grasping [1], [2], [3], [4], where the required grasp is predicted as a grasp rectangle calculated by choosing the best grasp from multiple grasp probabilities, our network generates three images from which we can infer grasp rectangles for multiple objects.”
High precision grasp pose detection in dense clutter (2016, methodology): “Using the same hand-labeled dataset, Redmon and Angelova pose grasp detection as a regression problem and solve it using convolutional neural network (CNN) methods [15].”
ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact (2019, background): “For example, [22], [23], [24] predict grasps for a parallel jaw gripper from a dataset of manually annotated images.”
Deep learning in robotics: a review of recent research (2017, background): “Challenge 6 (Sensor fusion) [77, 81, 83, 84, 86, 88] [87, 116, 117] [114] [116, 117]”
Helping Robots Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality Teleoperation (2020, methodology): “Learning often maps visual sensor data directly to grasps; the grasping rectangle describing suitable grasping in the image plane is a dominant representation [30], [31], [32], and mapping functions include logistic regression [30] and deep neural networks [31].”