First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations

By Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, Tae-Kyun Kim. 2017

563 citations recorded by the source corpus.

In this work we study the use of 3D hand poses to recognize first-person dynamic hand actions interacting with 3D objects. Towards this goal, we collected RGB-D video sequences comprised of more than 100K frames of 45 daily hand action categories, involving 26 different objects in several hand configurations. To obtain hand pose annotations, we used our own mo-cap system that automatically infers the 3D location of each of the 21 joints of a hand model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded the 6D object poses and provide 3D object models for a subset of hand-object interaction sequences. To the best of our knowledge, this is the first benchmark that enables the study of first-person hand actions with the use of 3D hand poses. We present an extensive experimental evaluation of RGB-D and pose-based action recognition by 18 baselines/state-of-the-art approaches. The impact of using appearance features, poses, and their combinations are measured, and the different training/testing protocols are evaluated. Finally, we assess how ready the 3D hand pose estimation field is when hands are severely occluded by objects in egocentric views and its influence on action recognition. From the results, we see clear benefits of using hand pose as a cue for action recognition compared to other data modalities. Our dataset and experiments can be of interest to communities of 3D hand pose estimation, 6D object pose, and robotics as well as action recognition.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Evaluation & Benchmarks, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Evaluation & Benchmarks

In Human-Computer Interaction

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Similar to the topics explored in early work, learning of visual representations capturing human hands, objects, and eye gaze has been extensively studied (Garcia-Hernando et al. 2018; Li, Liu, and Rehg 2018).”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “We do not compare with [15] since it only an-notates one hand (and note [11] has no hand boxes).”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “Due to the limited number of grasp types in the FHB dataset [19] and the HO-3D dataset [25], they are not suitable for training the generative model (see Appendix B).”
Human Hands as Probes for Interactive Object Understanding (2021, background): “…understanding can take different forms: grasp type classification [3,51,63] from a hand-defined taxonomy [15], hand keypoint and pose estimation [17], understanding gestures [18], detecting hands, their states and objects of interaction [55, 56], 3D reconstruction of the hand and the object of…”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, methodology): “Other works provide the position of the hand joints, obtained using manual annotations [3], data gloves [40] or wired sensors [20].”
FFNB: Forgetting-Free Neural Blocks for Deep Continual Learning (2021, methodology): “We evaluate the performance of our continual learning framework on the challenging task of action recognition, using the SBU and FPHA datasets [8, 58].”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “FPHA [17] (reported in [25]) 11.”
The Seventh Visual Object Tracking VOT2019 Challenge Results (2019, background): “A number of datasets in RGB+depth (RGBD) tracking focus on pedestrian and hand tracking [23, 85, 14, 30] and object pose estimation for robotics [13, 80].”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “Existing datasets rely on depth data [10, 16], visible marker equipment [14, 52] or multiple views [6, 44, 56, 68].”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “Our method doesn’t require training a hand pose network [20] to generate high accuracy on hand pose estimation under heavily occluded situations.”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, background): “Compared with depthbased 3D hand pose estimation datasets [4, 29, 33, 36, 44], existing RGB-based datasets [19,28,45,46] have very limited number of frames and subjects because obtaining accurate 3D annotation from RGB images is difficult.”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, background): “Second, there is a significant interest in training models to grasp 3D objects [4,63].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “H2O 571k ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ FPHA [26] 100k ✓ * (23k) · ✓ · ✓ ✓ ✓ · HOnnotate [32] 78k · ✓ · ✓ ✓ ✓ · ✓ ✓ Obman [34] 150k · ✓ · ✓ ✓ · · ✓ · Freihand [99] 37k · · · ✓ ✓ ✓ · · ✓ Panoptic [38] 1.”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, unknown): “We use the same subject split as [8], therefore, each object is present in both the training and test splits.”
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (2022, background): “Some existing works leverage magnetic sensors or mocap markers to track 3D hand poses and object poses [5, 18, 52].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Beyond object poses, datasets for joint estimation of the object and hand poses are also proposed [22, 88, 24, 42].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “The problem of 3D hand-object actions recognition has been addressed by the authors of [36], which released the Daily hand-object actions dataset containing 1175 videos belonging to 45 action categories.”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, methodology): “[17] using different thresholds for the 3D pose error.”