Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data

By Mohi Khansari, Daniel Kappler, Jianlan Luo, Jeff Bingham, Mrinal Kalakrishnan. 2020

26 citations recorded by the source corpus.

This paper introduces Action Image, a new grasp proposal representation that allows learning an end-to-end deep-grasping policy. Our model achieves 84% grasp success on 172 real world objects while being trained only in simulation on 48 objects with just naive domain randomization. Similar to computer vision problems, such as object detection, Action Image builds on the idea that object features are invariant to translation in image space. Therefore, grasp quality is invariant when evaluating the object-gripper relationship; a successful grasp for an object depends on its local context, but is independent of the surrounding environment. Action Image represents a grasp proposal as an image and uses a deep convolutional network to infer grasp quality. We show that by using an Action Image representation, trained networks are able to extract local, salient features of grasping tasks that generalize across different objects and environments. We show that this representation works on a variety of inputs, including color images (RGB), depth images (D), and combined color-depth (RGB-D). Our experimental results demonstrate that networks utilizing an Action Image representation exhibit strong domain transfer between training on simulated data and inference on real-world sensor streams. Finally, our experiments show that a network trained with Action Image improves grasp success (84% vs. 53%) over a baseline model with the same structure, but using actions encoded as vectors.

Subfields: Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Spatial equivariance has previously been shown to improve learning efficiency for vision-based picking [29, 30, 31, 32, 34].”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills and objects [42, 43, 44, 45].”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Some approaches to real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from expert demonstrations [27, 44, 45, 46, 47, 48].”
RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer (2020, background): “Images from a source domain can be modified at the pixel-level to resemble a target domain [4], [14].”