GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

By Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, Christian Theobalt. 2017

565 citations recorded by the source corpus.

We address the highly challenging problem of real-time 3D hand tracking based on a monocular RGB-only sequence. Our tracking method combines a convolutional neural network with a kinematic 3D hand model, such that it generalizes well to unseen data, is robust to occlusions and varying camera viewpoints, and leads to anatomically plausible as well as temporally smooth hand motions. For training our CNN we propose a novel approach for the synthetic generation of training data that is based on a geometrically consistent image-to-image translation network. To be more specific, we use a neural network that translates synthetic images to "real" images, such that the so-generated images follow the same statistical distribution as real-world hand images. For training this translation network we combine an adversarial loss and a cycle-consistency loss with a geometric consistency loss in order to preserve geometric properties (such as hand pose) during translation. We demonstrate that our hand tracking system outperforms the current state-of-the-art on challenging RGB-only footage.

Subfields: Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “nd renement. We propose a GAN based method in the second stage to rene the rendered 3DMM face pixels for higher realism as well as to inpaint the obfuscated head pixels around the rendered face. In [25,26], rendered images are modied to be more realistic by means of adversarial training. The generat”
Feature Boosting Network For 3D Pose Estimation (2019, neutral): “[43] 88.”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, methodology): “This hybrid approach has then been extended to work with RGB videos [36].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “PAN [21] 675K + + automatic 70 GAN [31] 300K + - synthetic FPHA [14] 100K + (23K frames) + automatic 26 (4 models) 6 ObMan [18] 150K + + - synthetic 2.”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, methodology): “is at the origin and the distance between wrist and mMCP is 1 [13].”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “Following [23], we evaluate our model on 2 se-”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, methodology): “[18] leveraged CycleGANs [45] to generate a “real” dataset”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “the proposed CMKG to other approaches [27, 31, 40, 59] on STB in Fig.”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Early approaches in modeling hand and object require the input of multi-view image [31] or RGB-D sequence [51].”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “, RGB images [2, 15, 32, 54, 60, 79, 80, 96] or RGB-D and depth sensors [24, 35, 48, 55, 57].”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, methodology): “Such data is created either by rendering synthetic datasets [5, 7, 23, 24, 38] or by capturing real datasets under controlled settings typically with little variation [8, 27, 32].”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “The approach [52] utilizes the video information to”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “Our method obtains state of-the-art results on standard benchmarks, even compared to methods using additional depth information in training [7, 47], camera intrinsics [27, 34], and post-processing optimization [27].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “Pioneering works of [17, 26, 40] have proposed ways to recover hand motion during object manipulation, yet without explicitly reasoning about the object pose.”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, unknown): “3D hand pose estimation has often been treated as predicting 3D positions of sparse joints [16, 32, 73].”
Metaverse: Perspectives from graphics, interactions and visualization (2022, background): “Mueller et al. combined a convolutional neural network with a kinematic 3D hand model, which addressed the highly challenging problem of real-time 3D hand tracking (Mueller et al., 2018).”