Cascaded Boundary Regression for Temporal Action Detection

By Jiyang Gao, Zhenheng Yang, Ramkant Nevatia. 2017

225 citations recorded by the source corpus.

Temporal action detection in long videos is an important problem. State-of-the-art methods address this problem by applying action classifiers on sliding windows. Although sliding windows may contain an identifiable portion of the actions, they may not necessarily cover the entire action instance, which would lead to inferior performance. We adapt a two-stage temporal action detection pipeline with Cascaded Boundary Regression (CBR) model. Class-agnostic proposals and specific actions are detected respectively in the first and the second stage. CBR uses temporal coordinate regression to refine the temporal boundaries of the sliding windows. The salient aspect of the refinement process is that, inside each stage, the temporal boundaries are adjusted in a cascaded way by feeding the refined windows back to the system for further boundary refinement. We test CBR on THUMOS-14 and TVSeries, and achieve state-of-the-art performance on both datasets. The performance gain is especially remarkable under high IoU thresholds, e.g. map@tIoU=0.5 on THUMOS-14 is improved from 19.0% to 31.0%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “[64], [65], [66] also copes with untrimmed videos.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, methodology): “To achieve this goal, various tasks have been proposed, such as temporal action localization [29, 10], action anticipation [11] and video prediction [32].”
Temporal Action Detection with Structured Segment Networks (2017, background): “CBR (Gao et al. 2017) begins with sliding windows and refines the boundaries progressively for finer localization.”
ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos (2020, unknown): “8 CBR [36] 60.”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, background): “Fully supervised temporal action localization has witnessed remarkable progress recently [39, 48, 15, 9, 37, 51, 47, 8, 2, 31].”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, methodology): “used two-stream features, but either did not perform fusion [16] or only tried the early fusion scheme [9, 15].”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization (2017, methodology): “(3) Although it uses a naive approach to detect actions, our method surpasses state-of-the-art approaches [3,16,46].”
Weakly-Supervised Action Localization With Background Modeling (2019, background): “More recent state-of-the-art methods [11, 10, 32, 5, 4] borrow intuitions from the recent object detection frameworks (e.”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “It is shown in the literature [9, 44] that TSN [40] surpasses C3D [34] features, and I3D [6] consistently outperforms TSN with different proposal methods.”
BMN: Boundary-Matching Network for Temporal Action Proposal Generation (2019, methodology): “In this work, we adopt twostream network [25] for feature encoding since it achieves great action recognition precision and is widely used in many video analysis methods [11, 19, 36].”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Beside the online action prediction task, the problem of temporal action detection [68, 73, 48, 33, 7, 53, 68, 14, 54, 16, 15, 65, 82] also copes with untrimmed videos.”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, unknown): “focused on temporal action localization [68, 38, 69, 56, 16].”
Graph Convolutional Networks for Temporal Action Localization (2019, background): “Temporal Action localization has attracted increasing attention in the last several years [6, 18, 26, 33, 34].”
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation (2018, methodology): “In our framework, we adopt two-stream network [33] as visual encoder, since this architecture has shown great performance in action recognition task [42] and has been widely adopted in temporal action detection and proposal generation tasks [49,26,12].”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, unknown): “6 - CBR [16] 50.”