Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

By Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, Li Fei-Fei. 2015

469 citations recorded by the source corpus.

Every moment counts in action recognition. A comprehensive understanding of human activity in video requires labeling every frame according to the actions occurring, placing multiple labels densely over a video sequence. To study this problem we extend the existing THUMOS dataset and introduce MultiTHUMOS, a new dataset of dense labels over unconstrained internet videos. Modeling multiple, dense labels benefits from temporal relations within and across classes. We define a novel variant of long short-term memory deep networks for modeling these temporal relations via multiple input and output connections. We show that this model improves action labeling accuracy and further enables deeper understanding tasks ranging from structured retrieval to action prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “Like we show for activity recognition, [67] demonstrates that including temporal information improves upon a single frame baseline.”
Generating Visual Explanations (2016, background): “Vision models which incorporate algorithms from reinforcement learning, speciﬁcally how to backpropagate through a sampling mechanism, have recently been applied to visual question answering [31] and activity detection [32].”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “[37] uses attention mechanism to densely label actions in a video sequence.”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Soft attention was proposed for image [9, 58] and video understanding [46, 48, 59] with spatial, temporal, and spatio-temporal variants.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “The availability of untrimmed video datasets (e.g. Chunhui et al., 2017; De Geest et al., 2016; Heilbron et al., 2015; Wan et al., 2016b; Yeung et al., 2017) have encouraged research and challenges/contests in motion recognition in this domain.”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “report a temporal recurrent attention model for dense labelling of videos [42].”
Action Recognition using Visual Attention (2015, background): “Yeung et al. (2015) do dense action labelling using a temporal attention based model on the input-output context and report higher accuracy and better understanding of temporal relationships in action videos.”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, background): “Most recently, Yeung et al. (2015) uses LSTM networks with multilabel outputs to recognize actions in videos.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “on, pointing to exciting future directions. Our dataset lends itself naturally to a variety of less explored tasks. We are planning to provide a wider set of challenges, including action localisation [50], video parsing [42], visual dialogue [7], goal completion [20] and skill determination [10] (e.”
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data (2016, background): “The fusion of neighboring frames within a sliding window with learned attention weights is proposed to enhance the performance of dense labeling of actions in RGB videos (Yeung et al. 2015).”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, methodology): “The core of our model builds upon Recurrent Neural Networks (RNN), which have been proved effective for capturing the temporal dependencies in data, and have been applied to challenging computer vision tasks including image captioning [7,8,17], video description [8,42,47], activity recognition [8,27”
Hierarchical Deep Temporal Models for Group Activity Recognition (2016, methodology): “In [50], a new dataset is introduced that contains dense multiple labels per frame for underlying action, and a novel Multi-LSTM is used to model the temporal relations between labels present in the dataset.”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, background): “Recently, many approaches [22, 9, 12, 17, 4] have been introduced to address this problem in fully-supervised setting, relying on the densely-annotated video data such as [8, 2].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Such findings hint that it is possible to predict when actions begin and end, and have inspired hundreds of Computer Vision datasets, models, and algorithms to study tasks like action recognition [19, 37, 71, 80, 81, 82].”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Dataset Recent years have witnessed an emergence of large-scale datasets containing sequences of common daily activities [42, 63, 60].”
What Actions are Needed for Understanding Human Actions in Videos? (2017, methodology): “Thus we choose Charades [31] as the best testbed for our analysis, and introduce MultiTHU-MOS [40], THUMOS [8], and ActivityNet [1] as needed to establish the generality of our conclusion.”