TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals

By Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, Ramkant Nevatia. 2017

486 citations recorded by the source corpus.

Temporal Action Proposal (TAP) generation is an important problem, as fast and accurate extraction of semantically important (e.g. human actions) segments from untrimmed videos is an important step for large-scale video analysis. We propose a novel Temporal Unit Regression Network (TURN) model. There are two salient aspects of TURN: (1) TURN jointly predicts action proposals and refines the temporal boundaries by temporal coordinate regression: (2) Fast computation is enabled by unit feature reuse: a long untrimmed video is decomposed into video units, which are reused as basic building blocks of temporal proposals. TURN outperforms the previous state-of-the-art methods under average recall (AR) by a large margin on THUMOS-14 and ActivityNet datasets, and runs at over 880 frames per second (FPS) on a TITAN X GPU. We further apply TURN as a proposal generation stage for existing temporal action localization pipelines, it outperforms state-of-the-art performance on THUMOS-14 and ActivityNet.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Beside the online action prediction task, the problem of temporal action detection [22], [56], [56], [57], [58], [59], [60], [61], [62], [63],”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[37] proposed a Temporal Unit Regression Network (TURN) model, that jointly predicts action proposals and refines the temporal boundaries by temporal coordinate regression using CNN.”
TALL: Temporal Activity Localization via Language Query (2017, background): “[5] proposed to use temporal coordinate regression to refine action boundary for temporal localization.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, methodology): “Temporal Unit Regression Network (TURN) [9] and Cascaded Boundary Regression (CBR) [10] exploit the temporal boundary regression mechanism for proposal generation and action detection.”
Weakly-Supervised Action Localization With Background Modeling (2019, background): “More recent state-of-the-art methods [11, 10, 32, 5, 4] borrow intuitions from the recent object detection frameworks (e.”
BMN: Boundary-Matching Network for Temporal Action Proposal Generation (2019, methodology): “For feature encoding, following previous works [18, 12], we adopt two-stream network [33] Table 2.”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, background): “For example, the TURN TAP system [14] is one of the fastest existing action proposal methods and yet, its computational cost exceeds by more than one order of magnitude that of our scheme.”
One-Shot Action Localization by Learning Sequence Matching Network (2018, background): “Action localization Fully supervised action localization problem has been extensively studied [25, 30, 38, 49, 35, 48, 15, 50, 51, 34, 23, 14, 28, 47, 8].”
What Do I Annotate Next? An Empirical Study of Active Learning for Action Localization (2018, background): “To unburden the computational requirements of sliding windows, a new line of work studies the use of action proposals to quickly scan a video in an attempt to reduce the search space [6,7,10,20,24,56].”
Attentive Moment Retrieval in Videos (2018, methodology): “Here we adopt the moment boundary adjustment strategy presented in [11].”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Dense video captioning lies at the intersection of event localization [25, 29, 33, 62, 65, 66, 84, 127] and event captioning [30,63,75,97,104].”
Human Action Recognition and Prediction: A Survey (2018, methodology): “Action anticipation aims to anticipate future actions from a history of actions (Gao et al., 2017).”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Beside the online action prediction task, the problem of temporal action detection [68, 73, 48, 33, 7, 53, 68, 14, 54, 16, 15, 65, 82] also copes with untrimmed videos.”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “These datasets have substantially advanced the progress of research on action proposal generation [17, 18, 5, 35] and action localization [59, 48, 61, 34, 9, 2, 4].”
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering (2019, background): “Temporal Unit Regression Network (TURN) [35] jointly predicts action proposals and refines the temporal boundaries”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “The above methods neglect the context information of proposals, and hence some attempts have been developed to incorporate the context to enhance the proposal feature [8, 17, 18, 52, 6].”
ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos (2020, unknown): “5 TURN [33] 54.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, methodology): “used two-stream features, but either did not perform fusion [16] or only tried the early fusion scheme [9, 15].”