Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection

By Sergey Levine, Peter Pastor, Alex Krizhevsky, Deirdre Quillen. 2016

2,254 citations recorded by the source corpus.

We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images independent of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. We describe two large-scale experiments that we conducted on two separate robotic platforms. In the first experiment, about 800,000 grasp attempts were collected over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and gripper wear and tear. In the second experiment, we used a different robotic platform and 8 robots to collect a dataset consisting of over 900,000 grasp attempts. The second robotic platform was used to test transfer between robots, and the degree to which data from a different set of robots can be used to aid learning. Our experimental results demonstrate that our approach achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. Our transfer experiment also illustrates that data from different robots can be combined to learn more reliable and effective grasping.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Andy Zeng
  5. Jitendra Malik
  6. Abhinav Kumar Gupta
  7. Vikash Kumar
  8. Dieter Fox

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Jitendra Malik
  3. Trevor Darrell
  4. Dacheng Tao
  5. Xiaogang Wang
  6. Li Fei-Fei
  7. Andrew Zisserman
  8. Ming-Hsuan Yang

In Control and Systems Engineering

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Dieter Fox
  5. Abhinav Kumar Gupta
  6. Vikash Kumar
  7. Silvio Savarese
  8. Abhishek Gupta

In Human-Computer Interaction

  1. Sergey Levine
  2. Abhinav Kumar Gupta
  3. Jitendra Malik
  4. Lerrel Pinto
  5. Dieter Fox
  6. Arsalan Mousavian
  7. Kristen Grauman
  8. Pieter Abbeel

In Robotics & Control

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Abhinav Kumar Gupta
  5. Silvio Savarese
  6. Dieter Fox
  7. Lerrel Pinto
  8. Vladlen Koltun

Citations in the corpus