SLAC: A Sparsely Labeled Dataset for Action Classification and Localization

By Hang Zhao, Zhicheng Yan, Heng Wang, Lorenzo Torresani, Antonio Torralba. 2017

31 citations recorded by the source corpus.

This paper describes a procedure for the creation of large-scale video datasets for action classification and localization from unconstrained, realistic web data. The scalability of the proposed procedure is demonstrated by building a novel video benchmark, named SLAC (Sparsely Labeled ACtions), consisting of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories. Using our proposed framework, annotating a clip takes merely 8.8 seconds on average. This represents a saving in labeling time of over 95% compared to the traditional procedure of manual trimming and localization of actions. Our approach dramatically reduces the amount of human labeling by automatically identifying hard clips, i.e., clips that contain coherent actions but lead to prediction disagreement between action classifiers. A human annotator can disambiguate whether such a clip truly contains the hypothesized action in a handful of seconds, thus generating labels for highly informative samples at little cost. We show that our large-scale dataset can be used to effectively pre-train action recognition models, significantly improving final metrics on smaller-scale benchmarks after fine-tuning. On Kinetics, UCF-101 and HMDB-51, models pre-trained on SLAC outperform baselines trained from scratch, by 2.0%, 20.1% and 35.4% in top-1 accuracy, respectively when RGB input is used. Furthermore, we introduce a simple procedure that leverages the sparse labels in SLAC to pre-train action localization models. On THUMOS14 and ActivityNet-v1.3, our localization model improves the mAP of baseline model by 8.6% and 2.5%, respectively.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [9, 19, 36, 7]) to train convolutional neural networks for video classification.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “Recently, datasets, such as TrecVid MED [29], Sports-1M [21], YouTube-8M [1], Something-something [12], SLAC [48], Moments in Time [28], and Kinetics [22] have focused on large-scale video classiﬁcation, often with automatically generated – and hence potentially noisy – annotations.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “This has been changing recently, with the release of the action classification benchmarks such as [10], [11], [12], [13], [14], [15].”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, background): “Researchers have devoted signiﬁcant and laudable efforts in creating video benchmarks of much larger size compared to the past [9, 12, 13, 14, 15, 16], both in terms of number of examples as well as number of action classes.”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, background): “Previous work has attempted to reduce this labeling effort through heuristics [51], but these methods still require a human annotator to clean up the final labels.”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, background): “Some work has attempted to reduce this labeling effort through heuristics [50] (although a human annotator is required to clean up the final labels) or by procuring weak labels in the form of accompanying meta data such as hashtags [12].”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “We report two sets of metrics: aggregate and per-class, which are equivalent to the class-agnostic and class-aware metrics in [12].”
Human Action Recognition and Prediction: A Survey (2018, background): “SLAC (Zhao et al., 2017) 2017 520,000 – 200 – RGB Uncontrolled”