Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge

By Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, Jianxiong Xiao. 2016

481 citations recorded by the source corpus.

Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC) [1]. A fully autonomous warehouse pick-and-place system requires robust vision that reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. In this paper we present an approach that leverages multiview RGB-D data and self-supervised, data-driven learning to overcome those difficulties. The approach was part of the MIT-Princeton Team system that took 3rd- and 4th-place in the stowing and picking tasks, respectively at APC 2016. In the proposed approach, we segment and label multiple views of a scene with a fully convolutional neural network, and then fit pre-scanned 3D object models to the resulting segmentation to get the 6D object pose. Training a deep neural network for segmentation typically requires a large amount of training data. We propose a self-supervised method to generate a large labeled dataset without tedious manual segmentation. We demonstrate that our system can reliably estimate the 6D pose of objects under a variety of scenarios. All code, data, and benchmarks are available at http://apc.cs.princeton.edu/

Subfields: Aerospace Engineering, Control and Systems Engineering, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Control and Systems Engineering

In Industrial and Manufacturing Engineering

Citations in the corpus

Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning (2018, methodology): “A common approach to deploying these methods in real systems involves pre-computing grasps from a database of known 3D object models [23], and indexing them at run-time with point cloud registration for object pose estimation [24], [25].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Considerable research in vision for manipulation has been devoted to object detectors and pose estimators [17, 18, 19, 20, 21].”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “An alternative approach is to use deep learning to estimate 3D object shape and pose directly from color and depth images [18, 59].”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “More recently, deep learning based approaches in computer vision are being adopted for the task of pose estimation of specific objects[33, 53, 54].”
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2017, background): “Several datasets and approaches have been introduced for the speciﬁc setting in the APC [14], [15].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Traditionally, perception for manipulation has centered around object detectors, segmentors, and pose estimators [21, 22, 23, 24, 25, 26].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Traditionally, methods in robot perception have used explicit “object” representations like instance segmentations, object classes, poses [18, 19, 20, 21, 22, 23].”
End-to-End Learnable Histogram Filters (2017, methodology): “…to learn from (Silver et al., 2016), training data for learning segmentation can be generated by simple algorithms such as background subtraction (Zeng et al., 2016), and reinforcement learning problems can be solved using training samples generated via trajectory optimization (Levine & Koltun,…”
PointNetGPD: Detecting Grasp Configurations from Point Sets (2018, background): “In [15], researchers introduced a pipeline for grasping objects from dense clutter.”
Self-supervised 6D Object Pose Estimation for Robot Manipulation (2019, methodology): “Previous works on self-supervised learning in robot perception mainly focus on learning object segmentation [6], object detection [7], and dense pixel-wise correspondences [29], [30], since it is relatively easy to obtain annotations in 2D image space.”
A survey on deep learning techniques for image and video semantic segmentation (2018, background): “Zeng et al.[112] present an object segmentation approach that leverages multi-view RGB-D data and deep learning techniques.”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “Classic vision-based grasping solutions often explicitly model contact forces with prior knowledge of object geometry, pose, and dynamics [8]–[11].”
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs (2019, background): “[22] proposed a self-supervised approach to semantic segmentation by utilizing the known background information.”
Deep Learning for 3D Point Clouds: A Survey (2019, unknown): “These datasets have further boosted the research of deep learning on 3D point clouds, with an increasingly number of methods being proposed to address various problems related to point cloud processing, including 3D shape classification, 3D object detection and tracking, 3D point cloud segmentation,”
Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching (2017, methodology): “r constructed from the same RGB-D data captured during recognition for novel objects or given by another system for known objects) are aligned to these visible surfaces via ICP-based pose estimation (Zeng et al. (2017)). To reduce the uncertainty and noise of these pose estimates, the placing primit”
Transfer of View-manifold Learning to Similarity Perception of Novel Objects (2017, background): “There has been a recent trend in exploring inherent information as supervisory signal, including using cycle consistency for learning dense correspondance(Zhou et al., 2015), camera motion for foreground segmentation(Zeng et al., 2016) and context information(Doersch et al., 2015).”
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation (2019, methodology): “Template matching techniques align 3D CAD models to observed 3D point clouds with algorithms such as iterative closest point [3, 51], or use hand crafted local descriptors to further guide the alignment process [25, 10].”
GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger (2018, background): “Our motivation stems from efforts in developing bin-picking systems to grasp novel objects in cluttered scenes and from the need to observe the geometry of contact to evaluate and control the quality of a grasp [5], [6].”