PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation

By Sida Peng, Yuan Liu, Qi-Xing Huang, Hujun Bao, Xiaowei Zhou. Computer Vision and Pattern Recognition, 2018

1,078 citations recorded by the source corpus.

This paper addresses the challenge of 6DoF pose estimation from a single RGB image under severe occlusion or truncation. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise vectors pointing to the keypoints and use these vectors to vote for keypoint locations. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occlusion LINEMOD and YCB-Video datasets by a large margin, while being efficient for real-time pose estimation. We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation. The code is available at https://zju3dv.github.io/pvnet/.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/743eab7fa743dc00532ea7c2bc0f6f8d87c93405

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “2 PVNet [Peng et al., 2019] 86.”
3D-FUTURE: 3D Furniture Shape with TextURE (2020, background): “In the unsupervised setting, 3D recovery has been recast as a 2D image reconstruction progress of unobserved views with diﬀerentiable rendering (Liu et al., 2019; Chen et al., 2019b).”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation (2020, methodology): “pose from RGB-D images in an end-to-end framework. The third category relies on establishing point correspondences. [2,12,17] regress the corresponding 3D object coordinate for each foreground pixel. [18,23,28] detect the keypoints of the object on image and then solve a Perspective-n-Point problem.”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, neutral): “Nonetheless, with the advent of deep learning, meth-odsbasedonneuralnetworksbecomedominantininstance-level objectposeestimation[16],[17],[18],[19],[20],[21].”
Deep Snake for Real-Time Instance Segmentation (2020, methodology): “An alternative method is to use standard CNNs to regress a pixel-wise vector field from the input image to guide the evolution of the initial contour [37, 33, 40].”
HybridPose: 6D Object Pose Estimation Under Hybrid Representations (2020, neutral): “Alternative keypoint representations include vector-fields [34] and patches [14].”
PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, methodology): “Instead, recent works utilized DNNs to detect 2D keypoints of an object, and computed 6D pose parameters with Perspective-n-Point (PnP) algorithms [37, 36, 41, 47].”
CosyPose: Consistent multi-view multi-object 6D pose estimation (2020, methodology): “Following [5,10,18], we evaluate on a subset of 2949 keyframes from videos of the 12 testing scenes.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, background): “However, the projection requires extrinsic camera parameters, which are challenging to estimate in most scenarios [24].”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “Some methods are now robust to partial occlusions [20, 36, 41], but many works rely on RGB-D data to handle this problem [5, 8, 22, 30], by fitting the 3D object model to depth data.”