Learning to Estimate 3D Hand Pose from Single RGB Images

By Christiane Zimmermann, Thomas Brox. 2017

819 citations recorded by the source corpus.

Low-cost consumer depth cameras and deep learning have enabled reasonable 3D hand pose estimation from single depth images. In this paper, we present an approach that estimates 3D hand pose from regular RGB images. This task has far more ambiguities due to the missing depth information. To this end, we propose a deep network that learns a network-implicit 3D articulation prior. Together with detected keypoints in the images, this network yields good estimates of the 3D pose. We introduce a large scale 3D hand pose dataset based on synthetic hand models for training the involved networks. Experiments on a variety of test sets, including one on sign language recognition, demonstrate the feasibility of 3D hand pose estimation on single color images.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Feature Boosting Network For 3D Pose Estimation (2019, methodology): “Different from the NYU Hand Pose dataset [41], which is mainly designed for hand pose estimation from depth images and the registered color images contain lots of artifacts, the large 3DHandPose dataset is highly suitable for 3D hand pose estimation from a single RGB image, as analyzed in [8].”
Understanding Human Hands in Contact at Internet Scale (2020, background): “ build the foundation for studying hands engaged in interaction with objects at Internet scale. Hand analysis is, of course, an area of long-standing interest in the ﬁeld with work on pose estimation [41,34], reconstruction [21,42], and grasp analysis [30,7]. These approaches, however, have largely ”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “We compare our hand module with the previous state-of-the-art hand approaches on three public hand benchmarks, STB [59], RHD [62] and MPII+NZSL [46].”
Contextual Attention for Hand Detection in the Wild (2019, background): “ColorHandPose [36] Specific poses 43,986 Synthetic HandNet [33] Ten subjects 212,928 Auto GTEA [21] Four subjects 663 Manual Oxford-Hand [24] Unconstrained 2686 Manual”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “[88] use a deep convolutional network for 3D hand pose estimation via a multi-stage approach.”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, background): “The field of 3D hand pose estimation has advanced rapidly, both in terms of accuracy [2, 4, 5, 6, 7, 26, 28, 32, 38, 45, 48, 49, 53, 54, 57, 60, 62] and dataset quality [10, 22, 43, 47, 52, 59].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”
Whole-Body Human Pose Estimation in the Wild (2020, background): “Most existing 2D RGB-based hand keypoint datasets are either synthetic [34,70] or captured in the lab environment [13,51,71].”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “Following the same condition used in [63, 5, 44], we assume that the global hand scale and the absolute depth of root joint are provided at test time.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “Several approaches for hand pose estimation from RGB images focus on regressing 3D skeleton joint positions [5, 16, 24, 40, 48, 51].”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “Realistic rendering and domain transfer can be used to train 3D pose estimation on synthetic images [31, 47, 73].”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, methodology): “Some works render synthetic hands for which the perfect ground truth is known [20, 63].”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “Zimmermann and Brox [59] first proposed a deep network to learn a network-implicit 3D articulation prior together with 2D key points for predicting 3D hand poses.”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “We also compare our hand skeleton (9e) to the 2D and 3D pose predictions of [68] (9f, 9g) and the 3D predictions of [47] (9h).”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, methodology): “The entire network is trained with a Rendered Hand Pose Dataset (RHD) created by [46] and a real-world dataset from Stereo Hand Pose Tracking Benchmark [43].”
WorkingHands: A Hand-Tool Assembly Dataset for Image Segmentation and Activity Mining (2019, methodology): “Hand image analysis datasets [2, 16, 21, 22, 38] were proposed for segmentation of hands, but many do not involve hand-object interactions or other body parts.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”
Reconstructing Hands in 3D with Transformers (2023, methodology): “For training data, we use multiple available sources of data with hand annotations, including both studio/controlled datasets with 3D ground truth [6, 19, 40, 56, 63, 64], and in-the-wild datasets annotated with 2D keypoint locations [15, 25, 52].”