HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization

By Hang Zhao, Antonio Torralba, Lorenzo Torresani, Zhicheng Yan. 2017

305 citations recorded by the source corpus.

This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos. We refer to it as HACS (Human Action Clips and Segments). We leverage consensus and disagreement among visual classifiers to automatically mine candidate short clips from unlabeled videos, which are subsequently validated by human annotators. The resulting dataset is dubbed HACS Clips. Through a separate process we also collect annotations defining action segment boundaries. This resulting dataset is called HACS Segments. Overall, HACS Clips consists of 1.5M annotated clips sampled from 504K untrimmed videos, and HACS Segments contains 139K action segments densely annotated in 50K untrimmed videos spanning 200 action categories. HACS Clips contains more labeled examples than any existing video benchmark. This renders our dataset both a large-scale action recognition benchmark and an excellent source for spatiotemporal feature learning. In our transfer learning experiments on three target datasets, HACS Clips outperforms Kinetics-600, Moments-In-Time and Sports1M as a pretraining source. On HACS Segments, we evaluate state-of-the-art methods of action proposal generation and action localization, and highlight the new challenges posed by our dense temporal annotations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, neutral): “ments in Time [43] 2019 YouTube 850 1 1M 3:00:0 313 ML-a mAP Something-Something [44] 2018 Scripted 234 1 220k 3:81:1 174 a, n, c Top-1/5 Acc. AVA [33] 2018 Film 108 1380 410k 2:73:5 80 ML-a mAP HACS [45] 2017 YouTube 861 2 1.5M 2 :00 200 a mAP Charades [46] 2016 Scripted 81 6.8 67k 12:89:3 157 ML-a”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Video understanding tasks, such as action recognition, have, for the most part, treated actions and activities as monolithic events [8, 38, 66, 87].”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “However, actions and activities are typically treated as monolithic events that occur in videos in action recognition [4, 25, 41, 60].”
Holistic Large Scale Video Understanding (2019, background): “In the last year also, a few other action datasets have been introduced with more samples, temporal duration and the diversity of category taxonomy, they are HACS [52], AVA [18], Charades [35] and Something-Something [17].”
Human Action Recognition and Prediction: A Survey (2018, background): “HACS Dataset (Zhao et al., 2019) is also a recently released large-scale dataset for action localization and recog-”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “Action labels are also a popular attribute of datasets for human action understanding that contains human-centric actions [60], [61], [62], [63], [64], [65], interaction [66], [67], [68], ﬁne-grained action understanding [63], [64], [65] and 3D data [69].”
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling (2019, background): “THUMOS 14 [17], ActivityNet [2], AVA [13], HACS [40]), video object detection ( e.g .”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, methodology): “More challenging datasets are proposed subsequently, including UCF101 [40], Kinetics [3], ActivityNet [11], Moments in Time [32], and others [25, 18, 50, 2, 52, 38, 33, 22, 46, 31, 20].”
Sensor-based and vision-based human activity recognition: A comprehensive survey (2020, background): “In 2019, two new RGB datasets were proposed, which contained a huge number of videos (HACS with 1,550,0 0 0 videos [115] and Moments in time with 1,0 0 0,0 0 0 videos [116] ).”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “…-6B almost consistently improves mAP Table 7: Finetuned temporal action localization results on THUMOS14 [Idrees et al., 2017], ActivityNet [Krishna et al., 2017], HACS Segment [Zhao et al., 2019] and FineAction [Liu et al., 2022]. with a notable margin from InternVideo2 -1B except in FineAction.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We evaluate VideoMAE and UniFormerV2 in InternVideo on Kinetics-400 [27], Kinetics-600 [69], Kinetics-700 [70], Something-in-Something-V1 [68], Something-in-Something-V2 [68], ActivityNet [79], HACS [80], and HMDB51 [81].”
Human Action Recognition From Various Data Modalities: A Review (2020, unknown): “Dataset Reference Year Modality #Class #Subject #Sample #View KTH [402] 2004 RGB 6 25 2,391 1 Weizmann [66] 2005 RGB 10 9 90 1 IXMAS [70] 2006 RGB 11 10 330 5 HDM05 [403] 2007 RGB,S 130 5 2,337 1 Hollywood [87] 2008 RGB 8 430 Hollywood2 [404] 2009 RGB 12 - 3,669 MSR-Action3D [254] 2010 S,D 20 10 567”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “This has been changing recently, with the release of the action classification benchmarks such as [10], [11], [12], [13], [14], [15], [16], [17], [18].”
BABEL: Bodies, Action and Behavior with English Labels (2021, unknown): “Existing datasets contain detailed action descriptions for only 2D videos, e.g., Activ-ityNet [28], AVA [11] and HACS [41].”