Understanding Human Hands in Contact at Internet Scale

By Dandan Shan, Jiaqi Geng, Michelle Shu, David F. Fouhey. 2020

405 citations recorded by the source corpus.

Hands are the central means by which humans manipulate their world and being able to reliably extract hand state information from Internet videos of humans engaged in their hands has the potential to pave the way to systems that can learn from petabytes of video data. This paper proposes steps towards this by inferring a rich representation of hands engaged in interaction method that includes: hand location, side, contact state, and a box around the object in contact. To support this effort, we gather a large-scale dataset of hands in contact with objects consisting of 131 days of footage as well as a 100K annotated hand-contact video frame dataset. The learned model on this dataset can serve as a foundation for hand-contact understanding in videos. We quantitatively evaluate it both on its own and in service of predicting and learning from 3D meshes of human hands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “Our work is unique in having large numbers of detailed segments for both hands and objects, distinguishing it from work that has boxes [35].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “This again shows that mis-classification of left/right hand is indeed a major issue that causes low mIoU in Table 1 for the models trained on [1,58].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[199] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “The second is handobject interactions from [37], trained on 100K images from YouTube along with 42K images from three egocentric datasets [1,39,40] of which 18K are from our videos [1].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, methodology): “For each frame f t in an egocentric video clip v , we use an off-the-shelf hand-object interaction model [55] that detects active objects—manipulated objects in contact with hands—resulting in a set of class-agnostic bounding boxes.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “[56] infer bounding boxes for hands in contact on the input RGB image.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Reconstructing hand and object jointly has been studied with both RGB input and RGB-D input [68–76].”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, neutral): “, 2022), 100 Days of Hands (100DOH) (Shan et al., 2020), Something-Something v2 (SS-V2) (Goyal et al.”
Masked Visual Pre-training for Motor Control (2022, methodology): “We construct the HOI data by combining Epic-Kitchens (Damen et al., 2018; 2021), Something-Something (Goyal et al., 2017b), and 100 Days of Hands (100-DOH) (Shan et al., 2020).”
Reconstructing Hands in 3D with Transformers (2023, methodology): “To encourage evaluation on in-the-wild images, we introduce a new dataset of annotations, HInt, by annotating hands from diverse image sources, including videos from YouTube [9, 51] and egocentric captures [12, 18].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “100DoH [77]: a large-scale dataset (131 days of recordings) consisting of egocentric and exocentric videos from YouTube.”
Detecting Hands and Recognizing Physical Contact in the Wild (2020, methodology): “% 57.41 % 33.37 % 41.76 % 59.54 % 58.93 % Table 2: Joint hand detection and contact recognition APs using different methods and datasets. M-RCNN denotes Mask-RCNN. 100DOH denotes video frames dataset [27] and C-Hands denotes our dataset ContactHands. ResNet-101 classiﬁers and evaluate contact recogn”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, methodology): “Baseline: We considered the Hand-Object Detector proposed in [53].”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “We then combine the Ego4d with ImageNet and the YouTube 100 Days of Hands dataset.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “We use the 100 Days of Hands (100DOH) hand-object detector [55] to predict hand and object bounding boxes.”
Human Hands as Probes for Interactive Object Understanding (2021, methodology): “As the detector from [55] only outputs boxes, we derive an approximate segmentation mask by pasting the detection boxes.”