Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

By Sven Bambach, Stefan Lee, David J. Crandall, Chen Yu. 2015

430 citations recorded by the source corpus.

Hands appear very often in egocentric video, and their appearance and pose give important cues about what people are doing and what they are paying attention to. But existing work in hand detection has made strong assumptions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Convolutional Neural Networks, and introduce a simple candidate region generation approach that outperforms existing techniques at a fraction of the computational cost. We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an application, we investigate the extent to which hand segmentation alone can distinguish between different activities. We evaluate these techniques on a new dataset of 48 first-person videos of people interacting in realistic environments, with pixel-level ground truth for over 15,000 hand instances.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “2022), or training them from scratch (Bambach et al. 2015) with first-person videos.”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “Most work with segments has focused only on hands [3] or has been gathered at the few thousand scale [14, 36].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “This again shows that mis-classification of left/right hand is indeed a major issue that causes low mIoU in Table 1 for the models trained on [1,58].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “53 [16] Sven Bambach, Stefan Lee, David J.”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “EgoHands fails on VLOG, getting 29.5 AP compared to 67.6 training on VLOG.”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “Here we show our results on 100DOH as well as generalizing (untrained) to VIVA [1], EgoHands [5], and VGG [25].”
Generalizing Hand Segmentation in Egocentric Videos With Uncertainty-Guided Model Adaptation (2020, methodology): “Bambach et al. [2] proposed a two-stage hand segmentation method by ﬁrst detecting hand bounding boxes with a convolutional neural network and then segmenting a hand region through Grabcut [40] in each detected bounding box.”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “Other studies tackled tasks related to EHOI recognition proposing hand-centric meth-ods [4, 1, 82, 52].”
3D Hand Shape and Pose From Images in the Wild (2019, neutral): “g.[23, 5]) as they are sensitive to occlusions.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “EgoHands [93]: This dataset is focuses on analysing hands in egocentric videos.”
EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition (2018, methodology): “For first-person view hand-related dataset, EgoHands [9], which is used for hand detection and segmentation, contains images captured by Google Glass with manually labeled pixel-”
Going Deeper into First-Person Activity Recognition (2016, positive): “[2] used hand regions to understand activity.”
Next-active-object prediction from egocentric videos (2017, methodology): “unconstrained egocentric videos is not trivial [48].”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “The traditional approach to create a RGB hand segmentation video dataset is through manual pixel-wise labeling [5, 7, 32, 50].”
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition (2019, neutral): “EgoHands [46] 4 48 RGB Head 130,000 4 2016 ! pixel-level annotation”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, background): “Many works provide egocentric RGB or RGB-D sequences for action recognition [3, 6, 7, 13, 48].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”
Detecting Hands and Recognizing Physical Contact in the Wild (2020, background): “f hands using their pose is not a robust approach. Some works consider hand-object interactions. Hasson et al. [12] propose to reconstruct hand and object meshes from monocular images. Bambach et al. [2] aim to locate hands interacting with an object, focusing on ﬁrst-person videos containing only t”
Embodied hands (2017, neutral): “However such cases can be tackled in future work inspired by works like [Ballan et al. 2012; Bambach et al. 2015; Oikonomidis et al. 2011b; Rogez et al. 2015; Sridhar et al. 2016; Tzionas et al. 2016; Tzionas and Gall 2015].”