HOnnotate: A Method for 3D Annotation of Hand and Object Poses

By Shreyas Hampali, Mahdi Rad, Markus Oberweger, Vincent Lepetit. Computer Vision and Pattern Recognition, 2019

527 citations recorded by the source corpus.

We propose a method for annotating images of a hand manipulating an object with the 3D poses of both the hand and the object, together with a dataset created using this method. Our motivation is the current lack of annotated real images for this problem, as estimating the 3D poses is challenging, mostly because of the mutual occlusions between the hand and the object. To tackle this challenge, we capture sequences with one or several RGB-D cameras and jointly optimize the 3D hand and object poses over all the frames simultaneously. This method allows us to automatically annotate each frame with accurate estimates of the poses, despite large mutual occlusions. With this method, we created HO-3D, the first markerless dataset of color images with 3D annotations for both the hand and object. This dataset is currently made of 77,558 frames, 68 sequences, 10 persons, and 10 objects. Using our dataset, we develop a single RGB image-based method to predict the hand pose when interacting with objects under severe occlusions and show it generalizes to objects not seen in the dataset.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/f1390ffb65d74c29eb4635d517d0cc17a984ad9d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “Improve hand and rigid objects with ARCTIC: We pre-train ArcticNet on ARCTIC, finetune on HO3D (hand + rigid objects).”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “Besides computing error in absolute 3D position, we also report errors after aligning the predictions with ground truths in post-processing [49, 50, 11].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Using this procedure, we annotate 500 images from the EPIC Kitchens [8] and the 100 Days HO3D [12] CP [2] GRAB [47] Ours”
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (2022, methodology): “Other works leverage carefully calibrated multi-camera systems [8, 40] or optimization algorithms [20] to ease the difficulty but are usually restricted to a third-person point of view.”
Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, methodology): “Then, we give ablation studies on the Figure 7: Visualization of generated grasps on in-domain objects from Obman dataset [19], and out-of-domain objects from HO-3D dataset [18].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Beyond object poses, datasets for joint estimation of the object and hand poses are also proposed [22, 88, 24, 42].”
Reconstructing Hands in 3D with Transformers (2023, methodology): “For training data, we use multiple available sources of data with hand annotations, including both studio/controlled datasets with 3D ground truth [6, 19, 40, 56, 63, 64], and in-the-wild datasets annotated with 2D keypoint locations [15, 25, 52].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Some methods assume no body visibility [63, 80, 98, 187, 188], while others assume partial observability by modifying cameras to capture the body [3, 57, 136, 161, 181].”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “To estimate human grasps from visual inputs, existing methods train models on large-scale datasets [7, 9, 31, 38, 83].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, methodology): “We evaluate our framework for joint 3D hand-object reconstruction and pose estimation on two recently released datasets: First Person Hand Action Benchmark [6] and HO3D [11] which provide pose annotations for all hand keypoints as well as the manipulated rigid object.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “[16] make use of a temporal consistency term when fitting hand-object configurations to RGB-D data.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “In this section, we provide detailed analyses on the FHB [87] and the HO-3D [95] datasets.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “H2O 571k ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ FPHA [26] 100k ✓ * (23k) · ✓ · ✓ ✓ ✓ · HOnnotate [32] 78k · ✓ · ✓ ✓ ✓ · ✓ ✓ Obman [34] 150k · ✓ · ✓ ✓ · · ✓ · Freihand [99] 37k · · · ✓ ✓ ✓ · · ✓ Panoptic [38] 1.”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “Following [23], this is measured through the discrepancy between 3D joints of the fitted MANO model and the ground truth 3D joints.”