Learning to Track for Spatio-Temporal Action Localization

By Philippe Weinzaepfel, Zaïd Harchaoui, Cordelia Schmid. 2015

348 citations recorded by the source corpus.

We propose an effective approach for spatio-temporal action localization in realistic videos. The approach first detects proposals at the frame-level and scores them with a combination of static and motion CNN features. It then tracks high-scoring proposals throughout the video using a tracking-by-detection approach. Our tracker relies simultaneously on instance-level and class-level detectors. The tracks are scored using a spatio-temporal motion histogram, a descriptor at the track level, in combination with the CNN features. Finally, we perform temporal localization of the action using a sliding-window approach at the track level. We present experimental results for spatio-temporal localization on the UCF-Sports, J-HMDB and UCF-101 action localization datasets, where our approach outperforms the state of the art with a margin of 15%, 7% and 12% respectively in mAP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Wu and Shao (2014) adopted deep belief networks (DBN) to model the distribution of skeleton joint locations and extract high-level features to represent humans at each frame in 3D space.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, neutral): “Following recent works on action detection [10, 48, 31], we report video-AP.”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, methodology): “The two-stream approach has recently been employed into several action recognition methods [4, 6, 7, 17, 25, 32, 35].”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, background): “It must learn to effectively utilize these observations, or frame glimpses, to reason on the temporal bounds of actions.”
Towards Weakly-Supervised Action Localization (2016, methodology): “Recently, signiﬁcant progress was achieved, see for example [5,6,7].”
VideoLSTM convolves, attends and flows for action recognition (2016, positive): “Following the previous works (van Gemert et al., 2015; Mettes et al., 2016; Weinzaepfel et al., 2015; Yu and Yuan, 2015), we use the first split and report mean average precision (mAP) over all 24 classes.”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “Spatio-temporal action localization is an active research area [12, 14, 17, 32, 40, 53].”
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences (2017, background): “motion features like 2D dense optical flow maps are precomputed and input to the networks [105, 97, 112, 34, 102, 122].”
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020, background): “There has been work that solely processes visual information to detect objects/actions/activity (Gaidon et al., 2013; Weinzaepfel et al., 2015; Shou et al., 2016; Dai et al., 2017; Shou et al., 2017).”
Action Tubelet Detector for Spatio-Temporal Action Localization (2017, methodology): “We compare our performance with late fusion of RGB+5flows when K = 6 to [9, 36], that use a twostream R-CNN, and to [34], which is based on actionness.”
Temporal Action Detection with Structured Segment Networks (2017, background): “y models the action structure via structural temporal pyramid pooling. By using sparse sampling, we further enable efﬁcient end-toend training. Note there are also works on spatial-temporal detection [14,55,26,51,32] and temporal video segmentation [16], which are beyond the scope of this paper. 3. ”
Human Action Localization with Sparse Spatial Supervision (2017, methodology): “Such annotations are, for example, used to train Convolutional Neural Networks (CNNs) [6], [7], [9], [10] at the bounding box level.”
AMTnet: Action-Micro-Tube Regression by End-to-end Trainable Deep Architecture (2017, methodology): “This dominant paradigm for action detection [6, 33, 20, 22], however, only provides a sub-optimal solution to the problem.”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “ is a challenging task with many potential applications in surveillance, autonomous driving, video description and search. To address this challenge, a number of successful methods have been proposed [11, 13, 18, 31, 36, 37, 45]. Such methods, however, typically rely on exhaustive supervision where ”
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos (2017, methodology): “Various methods have been developed, from the perspective of supervoxel merging [20, 54, 55], tracking [69, 42, 62, 52], object detection and linking [28, 14, 75, 42, 62], spatio-temporal segmentation [31, 70], and leveraging still images [21, 58, 22].”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, neutral): “[44] 63.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Motion cues play a crucial role in video understanding ﬁelds, such as action recognition [2, 5, 7, 39, 55, 57, 58], action detection [10, 62, 70, 74], human video generation [35, 37, 61], etc .”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, background): “Action recognition is a well studied problem with standard datasets [4, 8, 18, 28, 41, 44] focused on tasks such as classification [15, 42, 53, 58, 59] and temporal or spatiotemporal localization [25, 60, 62].”