CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos

By Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, Shih-Fu Chang. 2017

586 citations recorded by the source corpus.

Temporal action localization is an important yet challenging problem. Given a long, untrimmed video consisting of multiple action instances and complex background contents, we need not only to recognize their action categories, but also to localize the start time and end time of each instance. Many state-of-the-art systems use segment-level classifiers to select and rank proposal segments of pre-determined boundaries. However, a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. To this end, we design a novel Convolutional-De-Convolutional (CDC) network that places CDC filters on top of 3D ConvNets, which have been shown to be effective for abstracting action semantics but reduce the temporal length of the input data. The proposed CDC filter performs the required temporal upsampling and spatial downsampling operations simultaneously to predict actions at the frame-level granularity. It is unique in jointly modeling action semantics in space-time and fine-grained temporal dynamics. We train the CDC network in an end-to-end manner efficiently. Our model not only achieves superior performance in detecting actions in every frame, but also significantly boosts the precision of localizing temporal boundaries. Finally, the CDC network demonstrates a very high efficiency with the ability to process 500 frames per second on a single GPU server. Source code and trained models are available online at https://bitbucket.org/columbiadvmm/cdc.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Beside the online action prediction task, the problem of temporal action detection [22], [56], [56], [57], [58], [59], [60], [61], [62], [63],”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “To better localize temporal boundaries, a large body of work incorporated deep networks into the detection framework and obtained improved performance [5, 28, 10, 41, 2, 61, 42, 53, 59].”
ConvNet Architecture Search for Spatiotemporal Feature Learning (2017, background): “R-C3D [46] and CDC [31] achiving better performance on THUMOS’14.”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, methodology): “These datasets have substantially advanced the progress of research on action proposal generation [17, 18, 5, 35] and action localization [59, 48, 61, 34, 9, 2, 4].”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “Video understanding, especially action recognition, has been extensively studied for decades, where video representation learning serves as the fundamental problem of other video-related tasks, such as complex action recognition [23], action temporal localization [8, 41, 42], video caption [49, 52],”
To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression (2018, methodology): “Temporal coordinates of sentence queries are determined by applying the temporal boundary reﬁnement [ Shou et al. , 2017 ] on video attention weights.”
Diagnosing Error in Temporal Action Detectors (2018, background): “For example, [33] identiﬁes the problem of localization errors at high tIoU thresholds and devises the CDC network to predict actions at frame-level granularity.”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, unknown): “C3D 40.00 17.90 4.70 21.70 CDC [30] end2end C3D 45.30 26.00 0.20 23.80 R-C3D [41] end2end C3D 26.80 - - - TAL-Net [7] pre-extr.”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, background): “Note that [27] is the result of post-processing based on [41], making the comparison difficult.”
Temporal Context Network for Activity Localization in Videos (2017, methodology): “In contemporary work, Shou et al. [28] proposed a convolutional-de-convolutional (CDC) network by combing temporal upsampling and spatial downsampling for activity detection.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “Many recent CNNs for action classification [8, 52, 64, 60, 20], temporal action localization [66, 45, 65, 12, 70], and spatio-temporal action detection [19, 42, 41, 48, 4, 44, 62, 30, 74, 21, 26, 43] rely on pre-training a model on a large-scale dataset such as ImageNet or Kinetics and finetuning th”