Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation

By He Wang, Srinath Sridhar, Jingwei Huang, Julien P. C. Valentin, Shuran Song, Leonidas J. Guibas. 2019

913 citations recorded by the source corpus.

The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to ``instance-level'' 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce a \textbf{Normalized Object Coordinate Space (NOCS)}---a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new context-aware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Considerable research in vision for manipulation has been devoted to object detectors and pose estimators [17, 18, 19, 20, 21].”
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation (2019, background): “An interesting direction for future work is to train our keypoint detector using synthetic data, as demonstrated in [27,29].”
Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, methodology): “Existing approaches can be divided into two categories: supervised [2, 3, 6, 40] and unsupervised [8, 19, 24, 30, 42].”
Vector Neurons: A General Framework for SO(3)-Equivariant Networks (2021, methodology): “While both these methods [34, 19] need explicit 2D-to-3D supervision, relaxing supervision is possible by borrowing ideas from Transforming Auto-Encoders [14, 28].”
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (2022, methodology): “5 (5°5cm) on the NOCS [46] dataset where data is not captured during human-object interaction and does not suffer from heavy hand occlusions.”
Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly (2019, methodology): “Classic methods for assembly are often characterized by a perception module that first estimates the 6D object poses of observed parts [1], [2], [3], [4], [5], followed by a planner that then optimizes for picking and placing actions based on the inferred object poses and task-informed goal states.”
NASA: Neural Articulated Shape Approximation (2019, methodology): “Research has highlighted the importance of expressing information in a canonical frame [57], and one could interpret our method as a way to achieve this within the realm of articulated motion.”
Articulation-Aware Canonical Surface Mapping (2020, background): “Several previous approaches similarly pursue pixel to surface [2, 23, 24, 28, 42] or volume [35] mappings, but unlike our approach, crucially rely on direct supervision towards this end.”
GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, unknown): “In the future, we want to extend our work to more challenging scenarios, such as the lack of annotated real data [30] and unseen object categories or instances [83], [89].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “We note that in our method, our pose estimate is defined relative to a reference view of the object – this is in contrast to [3], which depends on a canonical pose definition – the subtleties of canonical pose definitions are discussed in [16], [44].”
Canonical Capsules: Unsupervised Capsules in Canonical Pose (2020, background): “The recently proposed CaSPR [39] fulfills this premise, but requires ground-truth canonical point clouds in the form of normalized object coordinate spaces [52] for supervision.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “, non-interactive perception [168]—for example, recognizing and localizing objects in a scene based on a camera image [49, 392, 411, 426].”
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (2023, methodology): “While category-level methods [5, 32, 54, 58, 69] remove these assumptions (instance-wise training and CAD models), they are limited to objects within the predefined category on which they are trained.”
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation (2020, methodology): “Hence, we choose [33] as our baseline for comparison.”
CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation (2019, methodology): “To build the segmentation-based baseline, we train a Mask-RCNN to provide object segmentation mask.”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “A recent dataset [41] was introduced for categorylevel 6D pose estimation, but the scenes are also static.”