3D Hand Shape and Pose From Images in the Wild

By Adnane Boukhayma, Rodrigo de Bem, Philip H. S. Torr. Computer Vision and Pattern Recognition, 2019

416 citations recorded by the source corpus.

We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/f9f420484ea9ec40f0eb4e2048798d3b0cbcb272

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, methodology): “Such data is created either by rendering synthetic datasets [5, 7, 23, 24, 38] or by capturing real datasets under controlled settings typically with little variation [8, 27, 32].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “[5] regress these parameters from an image and heatmaps obtained from OpenPose.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “[7] 13.”
Reconstructing Hands in 3D with Transformers (2023, methodology): “The earlier efforts [1, 2, 62] take inspiration from related work on human mesh recovery [27] - they use the MANO parametric hand model [48] and regress the hand pose and shape parameters given an RGB image as input.”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “Note that some other works [4, 12, 56] aim to predict the 3D hand shape other than hand joint locations, which are with different research targets compared with ours.”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, methodology): “f them rely on regressing parameters of the MANO model whereas our hand model is represented as a lightweight generator trained end-to-end in an unsupervised manner. The ﬁrst work by Boukhayma et al. [7] proposes a system analogous to the Human Mesh Recovery and is evaluated on images in the wild. T”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, background): “Other works [51, 1, 2] directly regress joint angles from the RGB image.”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “Similarly, there have been noticeable achievements in 3D hand pose estimation from a depth input [34, 48, 45, 47, 53, 58] or a single RGB image [62, 9, 15, 8, 11, 61].”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “A common solution when using MANO model is to optimize in the PCA space of 3D joint angles with an 2 regularizer [4, 74] for pose coefficients.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, background): “Note that direct supervising MANO parameters on synthetic dataset may obtain better performance [4].”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, background): “This problem has been studied for years, either taking as input RGB-D [63, 13, 47, 70, 26, 32] or RGB images [71, 49, 46, 61, 31, 2, 5, 21].”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “2 [5] Adnane Boukhayma, Rodrigo de Bem, and Philip H.”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “[5] trained their network that takes a single RGB image and estimates MANO parameters by minimizing the distance of the estimated hand joint locations and groundtruth.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, positive): “We ﬁnd it effective to regularize both hand pose and shape by applying (cid:96) 2 penalization as in [3].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “As in [4, 19, 54], we apply a mean square error term to the PCA hand components Lpca = ||θ||(2)2 to bias the estimated hand poses towards statistically plausible configurations.”