3D Hand Shape and Pose Estimation From a Single RGB Image

By Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, Junsong Yuan. Computer Vision and Pattern Recognition, 2019

504 citations recorded by the source corpus.

This work addresses a novel and challenging problem of estimating the full 3D hand shape and pose from a single RGB image. Most current methods in 3D hand analysis from monocular RGB images only focus on estimating the 3D locations of hand keypoints, which cannot fully express the 3D shape of hand. In contrast, we propose a Graph Convolutional Neural Network (Graph CNN) based method to reconstruct a full 3D mesh of hand surface that contains richer information of both 3D hand shape and pose. To train networks with full supervision, we create a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. When fine-tuning the networks on real-world datasets without 3D ground truth, we propose a weakly-supervised approach by leveraging the depth map as a weak supervision in training. Through extensive evaluations on our proposed new datasets and two public datasets, we show that our proposed method can produce accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose estimation accuracy when compared with state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/e3fc7fe51eae0ddb2435809740496fdaca86cd3c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “[9] proposed a graph convolution-based network which directly estimates vertices of hand mesh.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “[15] adopted a GraphCNN to estimate vertices of hand mesh.”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “These data together with their paired RGB images can be collected by corresponding devices or synthesized using pre-defined hand shape models [12, 37].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, methodology): “Some methods [5, 32, 9] estimated the low-dimensional parameters for a 3D deformable hand model [19] to fit the RGB inputs in order to retrieve the 3D canonical hand poses.”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “The dataset has been solved in recent publications [17] with a high error tolerance due to inaccurate ground truth annotations.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Many existing works [3, 12, 17, 29, 54, 53, 15, 14] rely on effective 2D pose estimation frameworks to first localize the 2D keypoints on the image plane, and then lift 3D poses from the estimated 2D joint positions.”
Reconstructing Hands in 3D with Transformers (2023, background): “Fol-lowup work [10, 16, 31, 39] follows a non-parametric approach and directly regresses the vertices of the MANO mesh.”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, background): “Similarly, there have been noticeable achievements in 3D hand pose estimation from a depth input [34, 48, 45, 47, 53, 58] or a single RGB image [62, 9, 15, 8, 11, 61].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “A number of recent methods plug MANO into end-to-end deep learning frameworks, obtaining accurate hand 3D shape and pose from single RGB images [3, 7, 53].”
MediaPipe Hands: On-device Real-time Hand Tracking (2020, background): “Other solutions are not lightweight enough to run real-time on commodity mobile devices[5] and thus are limited to platforms equipped with powerful processors.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, methodology): “Such data is created either by rendering synthetic datasets [4, 6, 19, 20, 33] or by capturing real datasets under controlled settings typically with little variation [7, 22, 27].”
Hand Gesture Recognition Based on Computer Vision: A Review of Techniques (2020, methodology): “A novel of estimating full 3D Where Graph Convolutional Neural Network (Graph CNN) utilized to reconstruct full 3D mesh for hand surface. hand shape and pose presented by Ge et al. [82] based on image.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “We also demonstrate superior robustness compared to [10], which we attribute to the combined training on 2D labeled in-the-wild images and the MoCap data.”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, background): “[15] is similar to ours but uses depth maps for supervision and focuses on hand pose estimation while our goal is to recover a realistic hand model.”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, methodology): “[21] use graph-CNNs to infer the 3D coordinates of a 1280-vertices hand mesh.”
Constraining dense hand surface tracking with elasticity (2020, background): “For recent, state-of-the-art results on dense hand tracking with a monocular system, we refer readers to [Ge et al. 2019].”