AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions

By Chunhui Gu, Chen Sun, Sudheendra Vijayanarasimhan, Caroline Pantofaru, David A. Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid and 1 more. 2017

1,208 citations recorded by the source corpus.

This paper introduces a video dataset of spatio-temporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 437 15-minute video clips, where actions are localized in space and time, resulting in 1.59M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video clips. AVA, with its realistic scene and action complexity, exposes the intrinsic difficulty of action recognition. To benchmark this, we present a novel approach for action localization that builds upon the current state-of-the-art methods, and demonstrates better performance on JHMDB and UCF101-24 categories. While setting a new state of the art on existing datasets, the overall results on AVA are low at 15.8% mAP, underscoring the need for developing new approaches for video understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

In Evaluation & Benchmarks

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Acquiring these labels can cost tens of thousands of hours of human time to annotate [23], and furthermore, when datasets become large, the beneﬁt of gathering more labels appears to diminish [27].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “1, 9 [16] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “25,26]. For example, Taskonomy [25] contains 26 tasks ranging from edge detection to vanishing point estimation and scene classication. In comparison, the number of tasks proposed for video datasets [1,5,27,28,29,30] remains modest. Often, this is limited by the source of videos in these datasets. Y”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “In fact, it is not even always clear what should be annotated, as human behavior exhibits a natural hierarchy, ranging from atomic actions [7] to high-level goals.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “We use the same approach as described in [51] to incorporate temporal context information via 3D networks.”
Moments in Time Dataset: One Million Videos for Event Understanding (2018, background): “ActivityNet [8] explores recognizing activities in video and AVA [15] explores recognizing amd localizing fine-grained actions.”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, background): “Researchers have devoted signiﬁcant and laudable efforts in creating video benchmarks of much larger size compared to the past [9, 12, 13, 14, 15, 16], both in terms of number of examples as well as number of action classes.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, background): “(1) The action detection datasets are comprised of untrimmed video samples, and the goal is to recognize and localize the action instances on temporal domain [23, 26] or spatial-temporal domain [21].”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “The contemporary effort of [13] also gathers data implicitly, but from films; its data is complementary and has more human-human interaction and”