Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications

By Lingzhi Zhang, Shenghao Zhou, Simon Stent, Jianbo Shi. 2022

106 citations recorded by the source corpus.

. Egocentric videos offer fine-grained information for high-fidelity modeling of human behaviors. Hands and interacting objects are one crucial aspect of understanding a viewer’s behaviors and in-tentions. We provide a labeled dataset consisting of 11,243 egocentric images with per-pixel segmentation labels of hands and objects being interacted with during a diverse array of daily activities. Our dataset is the first to label detailed hand-object contact boundaries. We introduce a context-aware compositional data augmentation technique to adapt to out-of-distribution YouTube egocentric video. We show that our robust hand-object segmentation model and dataset can serve as a foundational tool to boost or enable several downstream vision applications, including hand state classification, video activity recognition, 3D mesh reconstruction of hand-object interactions, and video inpainting of hand-object foregrounds in egocentric videos. Dataset and code are available at: https://github.com/owenzlz/EgoHOS In this supplementary materials, we first describe the following details of this work: 1). details of video frame collection; 2). training details of segmentation network; 3). how we chose the quantity of data augmentation; 4). the performance of using 100-DOH followed by PointRend. Please note that we also provide results of per-frame hand-object segmentations and ”see-through hand” application in egocentric videos. Please check our ”.mp4” file in the supplementary materials for details.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

Citations in the corpus

Segment Anything (2023, background): “ADE20K[117] BBBC038v1[12] Cityscapes[25] DOORS[80] DRAM[24] EgoHOS[113] GTEA[34,63] Hypersim[86] IBD[17] iShape[111] LVIS[44] NDD20[100] NDISPark[22,23] OVIS[81] PPDLS[74] Plittersdorf[46] STREETS[91] TimberSeg[38] TrashCan[52] VISOR[28,27] WoodScape[112] PIDRay[104] ZeroWaste-f[6] Figure 8: Samples”