FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

By Christiane Zimmermann, Duygu Ceylan, Jimei Yang, Bryan C. Russell, Max Argus, Thomas Brox. 2019

514 citations recorded by the source corpus.

Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this real-world dataset, we propose an iterative, semi-automated `human-in-the-loop' approach, which includes hand fitting optimization to infer both the 3D pose and shape for each sample. We show that methods trained on our dataset consistently perform well when tested on other datasets. Moreover, the dataset allows us to train a network that predicts the full articulated hand shape from a single RGB image. The evaluation set can serve as a benchmark for articulated hand shape estimation.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

Understanding Human Hands in Contact at Internet Scale (2020, background): “Hand analysis is, of course, an area of long-standing interest in the ﬁeld with work on pose estimation [41, 34], reconstruction [21, 42], and grasp analysis [30, 7].”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “ground pixels are composited onto a random COCO [35] image. This investigation is complementary to recent work on image-based estimation of object geometry [20,68], object pose [18,60], and hand pose [23,53,57,65,69]. Contact Representation: We observed in early experiments that the mean squared err”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, background): “Traintimeiswallclocktimeelapsedattheendofatrainingrun At RepMixer will lower the latency by 25.1% resolutions like 1024 1024, latency is lowered .9%.”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “[61, 12] introduces large-scale dataset with 3D annotation optimized from multi-view setups.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “H2O 571k ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ FPHA [26] 100k ✓ * (23k) · ✓ · ✓ ✓ ✓ · HOnnotate [32] 78k · ✓ · ✓ ✓ ✓ · ✓ ✓ Obman [34] 150k · ✓ · ✓ ✓ · · ✓ · Freihand [99] 37k · · · ✓ ✓ ✓ · · ✓ Panoptic [38] 1.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “l results show that the proposed Pose2Mesh outperforms the previous state-of-the-art 3D human pose and mesh estimation methods [26,30,31] on various publicly available 3D human body and hand datasets [21, 38, 67]. Particularly, our Pose2Mesh provides the state-of-the-art result on in-the-wild datase”
ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation (2020, background): “image classification [28], face recognition [24], fullbody [15], hand pose estimation [43] and multiview stereo reconstruction [29]), the gaze estimation community has so far relied on a heterogeneous environment where many papers employ custom data pre-processing and evaluation protocols, rendering”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “[20] Mean Shape [11]”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, background): “[47] captured a large-scale single hand pose and mesh dataset.”
Detecting Hands and Recognizing Physical Contact in the Wild (2020, methodology): “There are prior works for hand detection [3, 6, 7, 15–17, 19, 21, 22, 26, 28, 33, 35], hand pose estimation [12, 25, 29, 37, 38], and hand-tracking [3].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, methodology): “For RGB-based hand pose estimation, [20, 38] performed extensive manual annotation and provided a decent amount of labeled real-world instances.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Existing datasets rely on depth data [10, 16], visible marker equipment [14, 52] or multiple views [6, 44, 56, 68].”
Whole-Body Human Pose Estimation in the Wild (2020, background): “MPII [3] 25K 16 X X * X 40K MPII-TRB [10] 25K 40 X X * X 40K CrowdPose [30] 20K 14 X X X 80K PoseTrack [2] 23K 15 X X X 150K AI Challenger [62] 300K 14 X X X 700K COCO [31] 200K 17 X X * X 250K OneHand10K [60] 10K 21 X X X SynthHand [35] 63K 21 X X RHD [70] 41K 21 X X FreiHand [71] 130K 21 X MHP [13”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “renes them using a graph convolutional network on a mesh. Atlas 5 Back projecting image features into a voxel volume and then rening them using a 3D CNN has also been used for human pose estimation [25,50]. These works regress 3D heat maps that are used to localize joint locations. Deep Voxels [40] ”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, background): “Alternatively, studies showed good performance when utilizing hand pose estimation and mesh models to segment hands in frames [45, 62].”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “Wrist Thumb Index Middle Ring Pinky Output Figure 5: Qualitative results of our method on FreiHAND test set.”