Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

By Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield. Conference on Robot Learning, 2018

755 citations recorded by the source corpus.

Using synthetic data for training deep neural networks for robotic manipulation holds the promise of an almost unlimited amount of pre-labeled training data, generated safely out of harm's way. One of the key challenges of synthetic data, to date, has been to bridge the so-called reality gap, so that networks trained on synthetic data operate correctly when exposed to real-world data. We explore the reality gap in the context of 6-DoF pose estimation of known objects from a single RGB image. We show that for this problem the reality gap can be successfully spanned by a simple combination of domain randomized and photorealistic data. Using synthetic data generated in this manner, we introduce a one-shot deep neural network that is able to perform competitively against a state-of-the-art network trained on a combination of real and synthetic data. To our knowledge, this is the first deep network trained only on synthetic data that is able to achieve state-of-the-art performance on 6-DoF object pose estimation. Our network also generalizes better to novel environments including extreme lighting conditions, for which we show qualitative results. Using this network we demonstrate a real-time system estimating object poses with sufficient accuracy for real-world semantic grasping of known household objects in clutter by a real robot.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/3ac59cc39f4962b85eedcdc81d5126f4704563ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, positive): “Learning-Based Object Pose Estimator Xingyu Liu , Ruida Zhang , Chenyangguang Zhang , Gu Wang , Jiwen Tang , Zhigang Li , and Xiangyang Ji , Member, IEEE Abstract —6D pose estimation of rigid objects is a long-standing and challenging task in computer vision.”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “We use DOPE [33] to estimate the poses of these objects for our real-world experiments.”
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation (2019, methodology): “peline in Sec. III-B requires human annotation, although 3D reconstruction alleviates this manual labor. In the future, we plan to train our keypoint detector using synthetic data, as demonstrated in [23], [25]. Representing the robot action with a rigid transformation Taction is valid for robotic p”
Self-supervised 6D Object Pose Estimation for Robot Manipulation (2019, background): “A few recent methods show that it is possible to train exclusively with synthetic data and apply the trained network to the real world [8], [20].”
PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, background): “It is an important component in many real-world applications, such as robotic grasping and manipulation [6, 48, 55], autonomous driving [11, 5, 53], augmented reality [31] and so on.”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “Recent approaches based on deep learning have proposed to 1) directly estimate objects pose using CNN-based architectures [36, 44, 50] or 2) estimate 2D keypoints [32, 40, 42, 43] and solve for pose using the PnP-RANSAC algorithm.”
Deep Template-based Object Instance Detection (2021, background): “[38] showed that using different kinds of synthetic images reduced the performance gap between synthetic and real images.”
Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation (2020, background): “Consequently, they tend to segment every thing in an image and objects are usually over-segmented due to textures or concave shapes.”
FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation (2021, neutral): “6D Pose Estimation is an important component in lots of real-world applications, such as augmented reality [40], autonomous driving [13, 8, 69] and robotic grasping [9, 62, 17].”
CosyPose: Consistent multi-view multi-object 6D pose estimation (2020, methodology): “Object pose estimation is one of the oldest computer vision problems [1–3], yet it remains an active area of research [4–11].”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, result): “This investigation is complementary to recent work on image-based estimation of object geometry [20,68], object pose [18,60], and hand pose [23,53,57,65,69].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “For DOPE, we train models for an additional 7 objects besides the 5 provided.”
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (2023, methodology): “The object pose is often solved by direct regression [35, 67], or constructing 2D-3D correspondences followed by P n P [47, 55], or 3D-3D correspondences followed by least squares fitting [19, 20].”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, positive): “A common protocol estimates the 6-DoF object pose, followed by modelbased grasp planning [51], [30], [23], [48].”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “Many methods employ some level of rendering randomization [59], [60], [61], [9], [62], [63], including lighting conditions and textures.”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “The first strategy is to utilize simulated environments to generate virtual data [Tremblay et al., 2018].”