Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs

By Liuhao Ge, Hui Liang, Junsong Yuan, Daniel Thalmann. Computer Vision and Pattern Recognition, 2016

297 citations recorded by the source corpus.

Articulated hand pose estimation plays an important role in human-computer interaction. Despite the recent progress, the accuracy of existing methods is still not satisfactory, partially due to the difficulty of embedded high-dimensional and non-linear regression problem. Different from the existing discriminative methods that regress for the hand pose with a single depth image, we propose to first project the query depth image onto three orthogonal planes and utilize these multi-view projections to regress for 2D heat-maps which estimate the joint positions on each plane. These multi-view heat-maps are then fused to produce final 3D hand pose estimation with learned pose priors. Experiments show that the proposed method largely outperforms state-of-the-art on a challenging dataset. Moreover, a cross-dataset experiment also demonstrates the good generalization ability of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/a106bb2fa283d555255a2eec9f7a8520cabef95e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

End-to-End 3D Face Reconstruction with Deep Neural Networks (2017, background): “Three-dimensional information, being a strong prior invariant to view perspectives, has been demonstrated beneficial in different computer vision applications [29, 9, 11, 17, 18, 15].”
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor (2017, methodology): “Recent deep learning methods promise large learning capacities for hand pose estimation [39, 4, 29, 48, 46, 18].”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, methodology): “[13] project the depth image onto three orthogonal planes and train a 2D CNN for each projection, then fusing the results.”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “As a result, the majority of state-of-the-art methods apply deep learning methods to depth images [5, 6, 7, 8, 14, 18, 19, 32, 33].”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, background): “Numerous follow-up works have been proposed to improve the generative component [44, 46, 51, 52], and the learning-based discriminator [58, 16, 45, 49, 55, 7, 37, 22, 61, 5, 4, 29].”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, background): “Although 3D hand pose estimation with depth cameras [13, 7, 26, 41, 6] has gained tremendous success in recent years, the advance in monocular RGB-based 3D hand pose estimation[46, 18, 27, 15], however, still remains limited.”
Cross-Modal Deep Variational Hand Pose Estimation (2018, background): “Exploiting the depth information more directly, it has also been proposed to convert depth images into 3D multi-views [6] or volumetric representations [7] before feeding them to a 3D CNN.”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “This is significantly more challenging than hand pose estimation when a depth sensor is also available [58, 47, 16, 38, 54].”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Many existing works [3, 12, 17, 29, 54, 53, 15, 14] rely on effective 2D pose estimation frameworks to first localize the 2D keypoints on the image plane, and then lift 3D poses from the estimated 2D joint positions.”
MediaPipe Hands: On-device Real-time Hand Tracking (2020, background): “A large portion of previous work requires specialized hardware, e.g . depth sensors [13][16][17][3][4].”
Deep Kinematic Pose Regression (2016, methodology): “In this work, we propose to directly incorporate the articulated object model into the deep neutral network learning, which is the dominant approach for object pose estimation nowadays, for hand [32,29,21,22,41,8] or human body[33,35,20,10,17,18,31,30,40].”
3D Hand Shape and Pose From Images in the Wild (2019, background): “There is a rich literature on 3D hand pose and reconstruction from depth [62, 66, 10, 36, 61, 11, 43, 45, 19, 20, 24, 30, 37, 48, 52, 53, 59, 64], image and depth [26, 32, 49, 28], stereo [33, 65, 41] and multiple images [8, 31, 50].”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, methodology): “[5] extended this method by estimating multi-view 2D heatmaps.”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, background): “[8] proposed to project the depth image onto three orthogonal planes and fuse the corresponding 2D joint locations for the final 3D joint locations.”
V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map (2017, methodology): “[9] extended this method by exploiting multi-view CNN to estimate 2D heatmaps for each view.”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, background): “Estimating 3D hand poses from depth maps has made great progress in the past few years [10, 11, 26, 32, 43].”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, background): “In the past decade, we have witnessed a rapid advance in 3D hand pose estimation from depth images [35, 52, 11, 14, 13, 61, 10, 15].”