First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset

By Sangpil Kim, Xiao Hu, Anirudh Vegesana, Karthik Ramani. 2020

11 citations recorded by the source corpus.

First-person-view videos of hands interacting with tools are widely used in the computer vision industry. However, creating a dataset with pixel-wise segmentation of hands is challenging since most videos are captured with fingertips occluded by the hand dor-sum and grasped tools. Current methods often rely on manually segmenting hands to create annotations, which is inefficient and costly. To relieve this challenge, we create a method that utilizes thermal information of hands for efficient pixel-wise hand segmentation to create a multi-modal activity video dataset. Our method is not affected by fingertip and joint occlusions and does not require hand pose ground truth. We show our method to be 24 times faster than the traditional polygon labeling method while maintaining high quality. With the segmentation method, we propose a multi-modal hand activity video dataset with 790 sequences and 401,765 frames of "hands using tools" videos captured by thermal and RGB-D cameras with hand segmentation data. We analyze multiple models for hand segmentation performance and benchmark four segmentation networks. We show that our multi-modal dataset with fusing Long-Wave InfraRed (LWIR) and RGB-D frames achieves 5% better hand IoU performance than using RGB frames.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Simon Stent
  2. Shenghao Zhou
  3. Lingzhi Zhang
  4. Jianbo Shi
  5. Jitendra Malik
  6. Luc Van Gool
  7. Andrew Zisserman
  8. Dacheng Tao

In Human-Computer Interaction

  1. Lingzhi Zhang
  2. Shenghao Zhou
  3. Simon Stent
  4. Jianbo Shi

Citations in the corpus