End-to-End Learning of Action Detection from Frame Glimpses in Videos

By Serena Yeung, Olga Russakovsky, Greg Mori, Li Fei-Fei. 2015

627 citations recorded by the source corpus.

In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our intuition is that the process of detecting actions is naturally one of observation and refinement: observing moments in video, and refining hypotheses about when an action is occurring. Based on this insight, we formulate our model as a recurrent neural network-based agent that interacts with a video over time. The agent observes video frames and decides both where to look next and when to emit a prediction. Since backpropagation is not adequate in this non-differentiable setting, we use REINFORCE to learn the agent's decision policy. Our model achieves state-of-the-art results on the THUMOS'14 and ActivityNet datasets while observing only a fraction (2% or less) of the video frames.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Automatic Learning of Procedures From Web Instructional Videos (2017, background): “…remains an intensely studied problem-space, e.g., action recognition (Donahue et al. 2015; Wang et al. 2016), action detection (Singh et al. 2016; Yeung et al. 2016; Shou, Wang, and Chang 2016) and action labeling (Kuehne, Richard, and Gall 2017; Huang, Fei-Fei, and Niebles 2016; Bojanowski et…”
Dynamic Neural Networks: A Survey (2021, background): “discrete actions for dynamic inference graphs [15], [45], [69] or spatial/temporal sampling strategies [149], [151], [168], [176].”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Beside the online action prediction task, the problem of temporal action detection [22], [56], [56], [57], [58], [59], [60], [61], [62], [63],”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Similar hard attention was used in multiple object recognition [2], object localization [7, 38, 23], saliency map generation [27], and action detection [60].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[168] proposed an end-to-end training model that is formulated as a recurrent neural network-based agent.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “They have been successfully applied to language modeling [15], RGB based video analysis [16], [17], [18], [19], [20], [21], [22], [23], [24], and also skeleton based action recognition [9], [25], [26].”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “Yeung et al. report a temporal recurrent attention model for dense labelling of videos [43].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “As significant progress has been made on categorizing temporally trimmed video clips, recent research of human activity analysis is shifting towards a higher level understanding in real-world settings [11,22,29,35,43,49].”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “Most existing work on temporal action segmentation focuses on a fully supervised task [15, 26, 31, 19, 32, 22, 6, 37, 18, 36, 28].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Such findings hint that it is possible to predict when actions begin and end, and have inspired hundreds of Computer Vision datasets, models, and algorithms to study tasks like action recognition [19, 37, 71, 80, 81, 82].”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “Reinforcement learning has also been applied for dynamic feature prioritization in images [26] and video [45, 56], actively deciding which frames or image regions to visit next.”
Asynchronous Temporal Fields for Action Recognition (2016, methodology): “Moving towards more expressive deep networks such as LSTM has become a popular method for encoding such temporal information [48, 4, 65, 50, 58, 41, 64].”
Dynamic Computational Time for Visual Attention (2017, background): “Besides fine-grained recognition, attention models also work for various machine learning problems including machine translation [2], image captioning [64], image question answering [63, 12, 21, 65] and video activity recognition [66].”
Real-Time Online Action Detection Forests Using Spatio-Temporal Contexts (2016, background): “These new opportunities, however come with signiﬁcant challenges not encountered by the classical ofﬂine action recognition approaches [49, 42, 12, 40, 47, 44, 43, 39, 7, 15, 19, 50, 5, 10]: Realistic OAD applications require real-time processing that prevents the use of computationally expensive…”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “, 2019)) or selecting salient frames/clips conditioned on the input (Yeung et al., 2016; Wu et al., 2019b; Korbar et al., 2019; Gao et al., 2020).”
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (2017, neutral): “uses recurrent neural network-based models for dense activity recognition [66] and action detection from frame glimpses [67].”