Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching

By Andy Zeng, Shuran Song, Kuan-Ting Yu, E. Donlon, Francois Robert Hogan, Maria Bauzá, Daolin Ma, Orion Taylor, Melody Liu, E. Romo and 11 more. 2017

695 citations recorded by the source corpus.

This article presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses an object-agnostic grasping framework to map from visual observations to actions: inferring dense pixel-wise probability maps of the affordances for four different grasping primitive actions. It then executes the action with the highest affordance and recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional data collection or re-training. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT–Princeton Team system that took first place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu/

Subfields: Aerospace Engineering, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, methodology): “This pixel-wise parameterization of robot primitive actions, which we refer to as fully convolutional actionvalue functions [8], enables us to train effective pushing and grasping policies on a single robot arm in less than a few hours of robot time.”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “Our idea of spatially-invariant action space is inspired by a series of work by Zeng et al. and colleagues [34, 37, 39–42].”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, methodology): “As in [34, 33], we account for different grasping angles by rotating the input heightmap by 16 orientations (multiples of 22.”
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation (2019, methodology): “model-free methods [27], [5], [14], [8] directly evaluate the grasp quality from raw sensor inputs.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “In robotics, Zeng et. al [54] and Jang et. al [55] have proposed two-stream pipelines for affordance predictions of novel objects.”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, methodology): “We adapted a state-of-the-art grasping algorithm for our experiment [55, 54, 56], which consists of a convolutional network that predicts the probability of picking success for a scripted grasping primitive across a dense pixel-wise sampling of end effector locations and orientations across the…”
Learning Visual Affordances for Robotic Manipulation (2019, methodology): “1: Simulation Results on Random Arrangements (Mean %) Method Completion Grasp Success Action Efficiency Grasping-only [125] 90.”
Learning ambidextrous robot grasping policies (2019, methodology): “To compare with hand-coded methods used in practice, we implemented a best-effort suction-only policy and an ambidextrous policy based on geometric heuristics similar to those used in the Amazon Robotics Challenge ( 29 , 30 , 32 ).”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “With human experts directly annotating the training data [37, 14] or controlling the robot via teleoperation [39, 28], the system can quickly obtain positive examples to speed up the training process.”
Spatial Action Maps for Mobile Manipulation (2020, methodology): “This is motivated by gains in performance observed in other domains – for example, image segmentation [14] has been shown to benefit from pixel-aligned input and output representations, while end-to-end robotic manipulation [21, 45, 44, 43] is significantly more sample efficient when using FCNs to p”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, methodology): “All of the prior methods make use of simple grippers in an open-loop control setting [40], [22], [23], [53], [55], [56], [19].”
PyRep: Bringing V-REP to Deep Robot Learning (2019, background): “Robotics has benefited greatly from this progress, with many robotics systems opting to use deep learning in many or all of the processing stages of a typical robotics pipeline [2, 3].”