Cross-Modal Deep Variational Hand Pose Estimation

By Adrian Spurr, Jie Song, Seonwook Park, Otmar Hilliges. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

309 citations recorded by the source corpus.

The human hand moves in complex and high-dimensional ways, making estimation of 3D hand pose configurations from images alone a challenging task. In this work we propose a method to learn a statistical hand model represented by a cross-modal trained latent space via a generative deep neural network. We derive an objective function from the variational lower bound of the VAE framework and jointly optimize the resulting cross-modal KL-divergence and the posterior reconstruction objective, naturally admitting a training regime that leads to a coherent latent space across multiple modalities such as RGB images, 2D keypoint detections or 3D hand configurations. Additionally, it grants a straightforward way of using semi-supervision. This latent space can be directly used to estimate 3D hand poses from RGB images, outperforming the state-of-the art in different settings. Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods. Finally, the model is fully generative and can synthesize consistent pairs of hand configurations across modalities. We evaluate our method on both RGB and depth datasets and analyze the latent space qualitatively.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/75a32edc0fffd795d893a21c52a5b1f312217087

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “[60] Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges.”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “We also compare our hand skeleton (9e) to the 2D and 3D pose predictions of [68] (9f, 9g) and the 3D predictions of [47] (9h).”
Understanding Human Hands in Contact at Internet Scale (2020, background): “A full survey of this literature is beyond the scope of this paper and we refer the reader to [41, 34].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, unknown): “[21] and Yang et al.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “, RGB images [2, 15, 32, 54, 60, 79, 80, 96] or RGB-D and depth sensors [24, 35, 48, 55, 57].”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, methodology): “nd RHP in Table 5. The table shows InterHand2.6M 13 methods GT S GT H EPE (STB) EPE (RHP) Zimm. et al. [46] 3 3 8.68 30.42 Chen et al. [3] 3 3 10.95 24.20 Yang et al. [41] 3 3 8.66 19.95 Spurr et al. [27] 3 3 8.56 19.73 Spurr et al. [27] 7 7 9.49 22.53 InterNet (ours) 7 7 7.95 20.89 Table 5: EPE com”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “Second, most previous methods focus on predicting 3D joint positions [48, 34, 17, 3, 54].”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “Following the same condition used in [63, 5, 44], we assume that the global hand scale and the absolute depth of root joint are provided at test time.”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, background): “This problem has been studied for years, either taking as input RGB-D [63, 13, 47, 70, 26, 32] or RGB images [71, 49, 46, 61, 31, 2, 5, 21].”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, background): “Fig 7 shows the comparisons with state-of-the-art methods [46, 43, 18, 27, 22] on both RHD and STB datasets.”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “Two hand pose estimation approaches [32, 25] stand out for being similar to ours in spirit.”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, unknown): “Our approach significantly outperforms state-of-the-art approaches [48, 53].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “In contrast, variational approaches learn a shared latent space by an alternated training between variational autoencoders [46], disentanglement of the latent representation [57], or the alignment of latent spaces [56].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “The problem is challenging with respect to thirdperson scenarios [58, 74], due to self-occlusions and the limited amount of training data available [37, 39, 51, 70].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “Several approaches for hand pose estimation from RGB images focus on regressing 3D skeleton joint positions [5, 17, 26, 44, 52, 55].”