What Actions are Needed for Understanding Human Actions in Videos?

By Gunnar A. Sigurdsson, Olga Russakovsky, Abhinav Kumar Gupta. 2017

137 citations recorded by the source corpus.

What is the right way to reason about human activities? What directions forward are most promising? In this work, we analyze the current state of human activity understanding in videos. The goal of this paper is to examine datasets, evaluation metrics, algorithms, and potential future directions. We look at the qualitative attributes that define activities such as pose variability, brevity, and density. The experiments consider multiple state-of-the-art algorithms and multiple datasets. The results demonstrate that while there is inherent ambiguity in the temporal extent of activities, current datasets still permit effective benchmarking. We discover that fine-grained understanding of objects and pose when combined with temporal reasoning is likely to yield substantial improvements in algorithmic accuracy. We present the many kinds of information that will be needed to achieve substantial gains in activity understanding: objects, verbs, intent, and sequential reasoning. The software and additional information will be made available to provide other researchers detailed diagnostics to understand their own algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Temporal Relational Reasoning in Videos (2017, background): “However, it remains diﬃcult due to the ambiguity of describing activities at appropriate timescales [1].”
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, background): “More recently, tools in the video domain (Alwassel et al., 2018) are looking into forms of dataset bias in activity recognition (Sigurdsson et al., 2017).”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “Due to the expensive cost of multiple-labeling, most of previous models [24, 28] are optimized using single-style annotations (which means each sample is labeled by one annotator), whereas the inherent uncertainty of event localization [29] is ignored.”
RESOUND: Towards Action Recognition Without Representation Bias (2018, background): “Recently, [15] analyzed action recognition by considering multiple datasets and algorithms and pointed out future directions for algorithm design.”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Collecting these boundaries is not only notoriously burdensome and expensive, but also potentially ambiguous and often arbitrary [21, 29, 7].”
ON THE EFFECTIVENESS OF TASK GRANULARITY FOR TRANSFER LEARNING (2018, background): “s and action classes, with control over the similarity and dierences of actions, which is needed for ne-grained corpora. Examples include the Something-Something dataset [1] and the Charades dataset [4]. The rst version of Something-Something [1] has 100;000 videos of humanobject interactions, compr”
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition (2018, background): “Especially for classiﬁcation tasks in video, where the annotation process is tedious and sometimes hard to agree upon (where an action begins and ends for example) [46].”
Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization (2017, background): “We choose to define the Turkers’ task this way because defining the action’s starting time is easier than defining its end time [36].”
Diagnosing Error in Temporal Action Detectors (2018, methodology): “After playing let-us-agree, the question naturally arises, can we precisely localize actions in time? To address this question, we followed [36] and designed an instance-wise procedure that helped us characterize the level of human agreement achieved after annotating temporal bounds of a given actio”
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference (2018, background): “Human pose-based activity recognition [37, 40] methods aggregate motion and appearance information along tracks of human body parts to recognize human activity.”
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation (2019, methodology): “We are inspired by previous work which aggregate ideas, methods and ﬁndings to provide a unify survey of a subﬁeld of computer vision [22, 38, 43, 21].”
Evolving Losses for Unsupervised Video Representation Learning (2020, background): “Simultaneously, videos are more expensive to collect and annotate than images, as they require additional, and often ambiguous, temporal annotations [34].”