Action Tubelet Detector for Spatio-Temporal Action Localization

By Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, Cordelia Schmid. 2017

352 citations recorded by the source corpus.

Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level that are then linked or tracked across time. In this paper, we leverage the temporal continuity of videos instead of operating at the frame level. We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, i.e., sequences of bounding boxes with associated scores. The same way state-of-the-art object detectors rely on anchor boxes, our ACT-detector is based on anchor cuboids. We build upon the SSD framework [19]. Convolutional features are extracted for each frame, while scores and regressions are based on the temporal stacking of these features, thus exploiting information from a sequence. Our experimental results show that leveraging sequences offrantes significantly improves detection performance over using individual frames. The gain of our tubelet detector can be explained by both more accurate scores and more precise localization. Our ACT-detector outperforms the state-of-the-art methods for frame-mAP and video-mAP on the J-HMDB [12] and UCF-101 [31] datasets, in particular at high overlap thresholds.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[63] extended the Single Shot MultiBox Detector (SSD) [84] framework from 2D to 3D by proposing an Action Tubelet detector.”
Video Action Transformer Network (2018, background): “Some recent work has looked at such ﬁne-grained video understanding [8, 17, 21, 37], but has largely been limited to small datasets like UCF-24 [37, 38] or JHMDB [19].”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “The detected tubelets/boxes can then be optionally linked to form full action tubes [12, 17, 20, 32, 36, 40].”
PoTion: Pose MoTion Representation for Action Recognition (2018, neutral): “In particular, two-stream approaches have shown promising results in different video understanding tasks such as video classification [12, 32, 43], action localization [20, 30] and video segmentation [18, 38].”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “When the action boundaries are available, all (or most of) the frames enclosed by the temporal bounds can be considered relevant to the action, and thus state-of-the-art methods randomly or uniformly select frames to represent the action and train a classifier [30, 12, 33, 6, 34, 16].”
Pattern Recognition (2015, positive): “Following with previous works [3,9,18] , we report the experiment results on the ﬁrst split.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “Method Backbone Inputs Pre-train JHMDB (all splits) ACT [30] VGG RGB+Flow ImageNet 65.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “7% ACT [41] 65.”
Recurrent Convolutional Strategies for Face Manipulation Detection in Videos (2019, background): “In both cases the core idea is that we want the recurrent model to take as input a face “tubelet” [22, 23], that is a sequence of spatio-temporal, aligned face crops across time.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, background): “Also note that besides temporal action localization, there also exists a large body of work on spatio-temporal action localization [19, 23, 40], which is beyond the scope of this paper.”
Unified Graph Structured Models for Video Understanding (2021, background): “Consequently, whilst initial approaches to spatio-temporal action detection involved extending 2D object detectors [38, 46] temporally [26, 51, 1, 65], current leading methods [14, 61, 71] on the AVA dataset [17] all explicitly model relationships, with approaches which we show can be interpreted…”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “Class-specific detections are temporally aggregated by a simpler online linker [18, 37] based on action scores and overlap.”