6-DoF object pose from semantic keypoints

By Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G. Derpanis, Kostas Daniilidis. IEEE International Conference on Robotics and Automation, 2017

424 citations recorded by the source corpus.

This paper presents a novel approach to estimating the continuous six degree of freedom (6-DoF) pose (3D translation and rotation) of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior work, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training image data. Furthermore, the approach can be applied to instance- and class-based pose recovery. Empirically, we show that the proposed approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios with a cluttered background. For class-based object pose estimation, state-of-the-art accuracy is shown on the large-scale PASCAL3D+ dataset.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/5dfdc1ed397deca1e8451d09b7699914020b20c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Segmentation-Driven 6D Object Pose Estimation (2018, background): “Over the years, much effort has been invested in designing local feature descriptors that are invariant to various transformations [27, 40, 41, 42, 34, 32], so that they can be matched more robustly [29, 30, 14].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “In those cases, a weak-perspective camera model is often applied to approximate the 3D-to-2D transformation for keypoint estimation [51,26], by changing Eq.”
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2017, methodology): “In feature-based methods, local features are extracted from either points of interest or every pixel in the image and matched to features on the 3D models to establish the 2D-3D correspondences, from which 6D poses can be recovered [3], [4], [13].”
PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation (2018, methodology): “Inspired by recent methods [29, 30, 36], we estimate the object pose using a two-stage pipeline: we first detect 2D object keypoints using CNNs and then compute 6D pose parameters using the PnP algorithm.”
HybridPose: 6D Object Pose Estimation Under Hybrid Representations (2020, background): “Pose estimation via intermediate representation is sensitive to outliers in predictions, which are introduced by occlusion and cluttered backgrounds [37, 32, 40].”
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion (2019, methodology): “Newer methods address the challenge by learning to predict the 2D keypoints [3, 22, 32, 34, 35] and solve the poses by PnP [10].”
SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control (2018, methodology): “work on object and camera pose estimation from RGB/D data using learning methods [17]–[19], most of which use some form of explicit supervision.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, methodology): “Furthermore we use a feedforward, single-shot camera and shape regression network, while in principle this is a problem that could require iterative model fitting techniques to align a 3D deformable model to 2D landmarks [31].”
Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene (2017, background): “For instance, researchers have predicted 3D object pose [23, 32], low-dimensional parametric shapes [8, 34], and surface normals [28].”
PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, methodology): “Instead, recent works utilized DNNs to detect 2D keypoints of an object, and computed 6D pose parameters with Perspective-n-Point (PnP) algorithms [37, 36, 41, 47].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “Recent approaches based on deep learning have proposed to 1) directly estimate objects pose using CNNbased architectures [32], [40], [46] or 2) estimate 2D keypoints [27], [35], [37], [38] and solve for pose using the PnP-RANSAC algorithm.”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, background): “For alternate approaches to related problems, see [33, 34, 35, 36, 37].”
3D Bounding Box Estimation Using Deep Learning and Geometry (2016, methodology): “Table 2 shows that MultiBin modules are more effective than discretized classification [21] and also keypoint based method of [14] which is based on localizing keypoints and solving a sophisticated optimization to recover the pose.”
Real-time Deep Pose Estimation with Geodesic Loss for Image-to-Template Rigid Registration (2018, methodology): “to predict keypoints as models and then use object models to find the orientation [17], [18]; and 2) models that predict the pose of the object directly from images [19], [20].”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, methodology): “Deep-learning methods for viewpoint estimation follow roughly three different paths: direct estimation of Euler angles [55, 50, 33, 24, 62, 63], template-based matching [20, 32, 51], and keypoint detection relying on 3D bounding box corners [39, 52, 13, 34, 36] or semantic keypoints [35, 65].”