Hand Pose Estimation via Latent 2.5D Heatmap Regression

By Pavlo Molchanov, Thomas Breuel, Juergen Gall, Jan Kautz. European Conference on Computer Vision, 2018

359 citations recorded by the source corpus.

Estimating the 3D pose of a hand is an essential part of human-computer interaction. Estimating 3D pose using depth or multi-view sensors has become easier with recent advances in computer vision, however, regressing pose from a single RGB image is much less straightforward. The main difficulty arises from the fact that 3D pose requires some form of depth estimates, which are ambiguous given only an RGB image. In this paper we propose a new method for 3D hand pose estimation from a monocular image through a novel 2.5D pose representation. Our new representation estimates pose up to a scaling factor, which can be estimated additionally if a prior of the hand size is given. We implicitly learn depth maps and heatmap distributions with a novel CNN architecture. Our system achieves state-of-the-art accuracy for 2D and 3D hand pose estimation on several challenging datasets in presence of severe occlusions.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/4975a215804aefac5765cb6e69cdaf9d9f3caec3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “3D pose estimation for hands [22, 31, 45, 56, 67] and objects [15, 26, 36, 39, 43, 49, 63, 64] have often been tackled in isolation [38].”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “Some methods use additional data, some of which not publicly available, for higher accuracy, including: synthetic images with ground truth hand mesh [10], depth images [48, 3, 34], real images with 2D annotations [17], and real images with 3D labels from a panoptic stereo [47].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, background): “Similarly, there have been noticeable achievements in 3D hand pose estimation from a depth input [34, 48, 45, 47, 53, 58] or a single RGB image [62, 9, 15, 8, 11, 61].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “Several approaches for hand pose estimation from RGB images focus on regressing 3D skeleton joint positions [5, 16, 24, 40, 48, 51].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, background): “Note that [10, 6] leverage on depth data (more information than segmentation since the background has infinite depth) for training and [9, 5] utilize a deformable hand model and additional datasets.”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “Following [37] and [23], we report our results using a PCK curve and the corresponding AUC in the right of Fig.”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “[15] cannot model uncertainty of the prediction in z-axis, while ours can.”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “Much of the progress made in hand pose estimation have focused on using depth image inputs [5, 6, 7, 8, 11, 14, 15, 18, 19, 32, 33, 35].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, positive): “Recent work obtains promising results for 2D and 3D hand pose estimation from monocular RGB images using convolutional neural networks [7, 16, 32, 40, 55, 56, 73].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, methodology): “e characterized by the individual actions a person takes in interaction with the surrounding objects and the environment. A signiﬁcant amount of research has focused on visual understanding of humans [8, 25,36,40,59,65,70,74] and objects [4,28,64,66], in isolation from each other. However, the probl”