Single Shot Temporal Action Detection

By Tianwei Lin, Xu Zhao, Zheng Shou. ACM Multimedia, 2017

483 citations recorded by the source corpus.

Temporal action detection is a very important yet challenging problem, since videos in real applications are usually long, untrimmed and contain multiple action instances. This problem requires not only recognizing action categories but also detecting start time and end time of each action instance. Many state-of-the-art methods adopt the "detection by classification" framework: first do proposal, and then classify proposals. The main drawback of this framework is that the boundaries of action instance proposals have been fixed during the classification step. To address this issue, we propose a novel Single Shot Action Detector (SSAD) network based on 1D temporal convolutional layers to skip the proposal generation step via directly detecting action instances in untrimmed video. On pursuit of designing a particular SSAD network that can work effectively for temporal action detection, we empirically search for the best network architecture of SSAD due to lacking existing models that can be directly adopted. Moreover, we investigate into input feature types and fusion strategies to further improve detection accuracy. We conduct extensive experiments on two challenging datasets: THUMOS 2014 and MEXaction2. When setting Intersection-over-Union threshold to 0.5 during evaluation, SSAD significantly outperforms other state-of-the-art systems by increasing mAP from $19.0%$ to $24.6%$ on THUMOS 2014 and from 7.4% to $11.0%$ on MEXaction2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b4fdbe0c627eb374fa9f347157ae94c5a2fcad67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, background): “In addition to the works reviewed above, there exist other classes of approaches, such as those based on single-shot detectors [2, 27] or reinforcement learning [52].”
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation (2018, methodology): “Most detection methods [32,34,49] take these two stages separately, while there is also method [26,3] taking these two stages jointly.”
Gaussian Temporal Awareness Networks for Action Localization (2019, methodology): “(2) Single Shot Action Detection (SSAD) [19] is the 1D variant version of Single Shot Detection [23], which generates action proposals by multiple temporal anchor layers.”
Multilevel Language and Vision Integration for Text-to-Clip Retrieval (2018, unknown): “SSAD (Lin, Zhao, and Shou 2017) performs single-shot temporal activity detection following the one-stage object detection method SSD (Liu et al. 2016).”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, methodology): “Inspired by the single-shot object detector [30] and its successful applications in temporal activity detection [59, 28], we apply a hierarchical convolutional network to directly produce multi-scale candidate moments from the input video stream.”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, methodology): “Other categories of methods have also been studied, such as those based on single-shot detectors [4, 30] or sequential decision-making process [48, 2].”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Third, there exist some approaches [18, 34] based on single-shot detection like SSD [21] for fast inference.”
To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression (2018, background): “Although promising results have been achieved [ Oneata et al. , 2013; Yeung et al. , 2016; Shou et al. , 2016; Gao et al. , 2017b; Lin et al. , 2017; Escorcia et al. , 2016 ] , one major limitation is that they are restricted to a predeﬁned list of action categories, which cannot precisely identify…”
BMN: Boundary-Matching Network for Temporal Action Proposal Generation (2019, methodology): “In this work, we adopt twostream network [25] for feature encoding since it achieves great action recognition precision and is widely used in many video analysis methods [11, 19, 36].”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “These datasets have substantially advanced the progress of research on action proposal generation [17, 18, 5, 35] and action localization [59, 48, 61, 34, 9, 2, 4].”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “[37] presented the first single-stage TAL using convolutional networks, borrowing ideas from a single-stage object detector [44].”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, background): “action recognition [31, 37, 23] and temporal action detection [50, 20].”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, background): “Others [28, 5, 27] tackle the problem using a single-stage model, where actions are detected directly.”
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language (2019, methodology): “Several related tasks are proposed for different scenarios, such as temporal action localization (Zhao et al. 2017), anomaly detection (Hasan et al. 2016), video summarization (Song et al. 2015; Chu, Song, and Jaimes 2015), and moment localization with nat-ural language (Gao et al. 2017; Hendricks…”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “Approaches on this task can be grouped into three categories: (1) methods performing frame or segment-level classification where the smoothing and merging steps are required to obtain the temporal boundaries [33, 28, 51]; (2) approaches employing a two-stage framework involving proposal generation, ”