Analysis of Hand Segmentation in the Wild

By Aisha Urooj Khan, Ali Borji. 2018

99 citations recorded by the source corpus.

A large number of works in egocentric vision have concentrated on action and object recognition. Detection and segmentation of hands in first-person videos, however, has less been explored. For many applications in this domain, it is necessary to accurately segment not only hands of the camera wearer but also the hands of others with whom he is interacting. Here, we take an in-depth look at the hand segmentation problem. In the quest for robust hand segmentation methods, we evaluated the performance of the state of the art semantic segmentation methods, off the shelf and fine-tuned, on existing datasets. We fine-tune RefineNet, a leading semantic segmentation method, for hand segmentation and find that it does much better than the best contenders. Existing hand segmentation datasets are collected in the laboratory settings. To overcome this limitation, we contribute by collecting two new datasets: a) EgoYouTube-Hands including egocentric videos containing hands in the wild, and b) HandOverFace to analyze the performance of our models in presence of similar appearance occlusions. We further explore whether conditional random fields can help refine generated hand segmentations. To demonstrate the benefit of accurate hand maps, we train a CNN for hand-based activity recognition and achieve higher accuracy when a CNN was trained using hand maps produced by the fine-tuned RefineNet. Finally, we annotate a subset of the EgoHands dataset for fine-grained action recognition and show that an accuracy of 58.6% can be achieved by just looking at a single hand pose which is much better than the chance level (12.5%).

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “The other two datasets EYTH [73] and EGTEA [33] provide only binary mask labels for both hands without differentiating between left or right.”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “Therefore, understanding hands with computer vision systems has been deeply explored through hand tracking [41, 54], hand pose estimation [18, 21, 22, 27, 29], grasp detection [10, 47], hand gesture recognition [58], multi-view prediction [29], and hand-action classiﬁcation [50].”
Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection (2020, methodology): “First, many datasets [10, 12, 13] only contain binary-label for hand segmentation, which does not distinguish between the left and right hand.”
WorkingHands: A Hand-Tool Assembly Dataset for Image Segmentation and Activity Mining (2019, methodology): “Yale [6] 0 18K N 11 Khan et al. [16] 1.6K N Man.”
Generalizing Hand Segmentation in Egocentric Videos With Uncertainty-Guided Model Adaptation (2020, methodology): “Recently, Urooj and Borji [43] used fully convolutional networks (ReﬁneNet-ResNet101 [35] originally proposed for semantic segmentation) for hand segmentation and achieved state-of-the-art performance.”