Temporal Action Detection with Structured Segment Networks

By Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, Dahua Lin. 2017

982 citations recorded by the source corpus.

This paper addresses an important and challenging task, namely detecting the temporal intervals of actions in untrimmed videos. Specifically, we present a framework called structured segment network (SSN). It is built on temporal proposals of actions. SSN models the temporal structure of each action instance via a structured temporal pyramid. On top of the pyramid, we further introduce a decomposed discriminative model comprising two classifiers, respectively for classifying actions and determining completeness. This allows the framework to effectively distinguish positive proposals from background or incomplete ones, thus leading to both accurate recognition and precise localization. These components are integrated into a unified network that can be efficiently trained in an end-to-end manner. Additionally, a simple yet effective temporal action proposal scheme, dubbed temporal actionness grouping is devised to generate high quality action proposals. We further study the importance of the decomposed discriminative model and discover a way to achieve similar accuracy using a single classifier, which is also complementary with the original SSN design. On two challenging benchmarks, THUMOS’14 and ActivityNet, our method remarkably outperforms previous state-of-the-art methods, demonstrating superior accuracy and strong adaptivity in handling actions with various temporal structures.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “[64], [65], [66] also copes with untrimmed videos.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[175] adopted explicit structural modeling in the temporal dimension.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “(3) R-C3D+TC and SSN+TC.”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “To study the impact of this hyperparameter, we perform experiments when #videos ∈ [10, 50, 100, 200], the results are shown in Table 9.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “Purely CNN based approaches such as structured segment networks [37] or temporal convolutional networks [18] have recently shown convincing results on several action segmentation benchmarks.”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, methodology): “Specifically, we experiment with two different proposal approaches: SSN [46] pre-trained on ActivityNet action proposal and a heuristic uniform proposal.”
Diagnosing Error in Temporal Action Detectors (2018, methodology): “Currently, researchers have appealing intuitions to tackle video action localization problem [1,10,34,45], they are equipped with large datasets to validate”
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language (2019, methodology): “Several related tasks are proposed for different scenarios, such as temporal action localization (Zhao et al. 2017), anomaly detection (Hasan et al.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, background): “Besides action classiﬁcation, some researchers recently have focused on temporal action localization [68, 38, 69, 56, 16].”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Second, proposal-based techniques [27, 31, 36, 38] extract all action proposals and refine their boundaries for action detection.”
Learning Salient Boundary Feature for Anchor-free Temporal Action Localization (2021, background): “designed SSN [44] in which course proposals are first divided into three semantic parts, learned”
Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “Generally speaking, there are two popular paradigms: the two-stage paradigm [9, 30, 39, 41, 60, 64, 71, 80] first detects classagnostic action proposals, which cover correct segments with high recall, then classifies and refines each of these proposals.”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “The above methods neglect the context information of proposals, and hence some attempts have been developed to incorporate the context to enhance the proposal feature [8, 17, 18, 52, 6].”
One-Shot Action Localization by Learning Sequence Matching Network (2018, background): “Action localization Fully supervised action localization problem has been extensively studied [25, 30, 38, 49, 35, 48, 15, 50, 51, 34, 23, 14, 28, 47, 8].”
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, background): “In general, human action can be recognized from multiple modalities(Simonyan and Zisserman 2014; Tran et al. 2015; Wang, Qiao, and Tang 2015; Wang et al. 2016; Zhao et al. 2017), such as appearance, depth, optical flows, and body skeletons (Du, Wang, and Wang 2015; Liu et al.”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Beside the online action prediction task, the problem of temporal action detection [68, 73, 48, 33, 7, 53, 68, 14, 54, 16, 15, 65, 82] also copes with untrimmed videos.”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, methodology): “Two action proposal generation methods are evaluated: Boundary Sensitive Network (BSN) [35] and Temporal Actionness Grouping (TAG) [61].”