HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

By Yunze Liu, Yun Liu, Zhoujie Fu, Kangbo Lyu, Weikang Wan, Hao Shen, Bo-Hua Liang, He Wang, Li Yi. Computer Vision and Pattern Recognition, 2022

379 citations recorded by the source corpus.

We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of category-level human-object interaction. HOI4D consists of 2.4M RGB-D egocentric video frames over 4000 sequences col-lected by 9 participants interacting with 800 different ob-ject instances from 16 categories over 610 different indoor rooms. Frame-wise annotations for panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action have also been provided, together with reconstructed object meshes and scene point clouds. With HOI4D, we establish three benchmarking tasks to pro-mote category-level HOI from 4D visual signals including semantic segmentation of 4D dynamic point cloud se-quences, category-level object pose tracking, and egocen-tric action segmentation with diverse interaction targets. In-depth analysis shows HOI4D poses great challenges to existing methods and produces huge research opportunities.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/2a0b8be3594e8163f9ea4988658223d7c46cfcb3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “Since most existing articulated object datasets contain neither twohands nor human bodies [41, 43, 76], and having a human in the scene is a realistic setting, we believe that ARCTIC brings additional challenges of heavy occlusion and dynamic manipulation to the depth community.”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “Therefore, we utilize several human-object interaction datasets, including datasets without depth information, such as Epic-Kitchens (Damen et al., 2018), and datasets with better 3D annotations, such as HOI4D (Liu et al., 2022).”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18, 19, 21, 40, 62, 67], to predict object information [30, 136] (RGB & 3D) [10], graphs [24] or environment information [28, 81] such as heatmaps [39, 80].”