R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

By Huijuan Xu, Abir Das, Kate Saenko. IEEE International Conference on Computer Vision, 2017

772 citations recorded by the source corpus.

We address the problem of activity detection in continuous, untrimmed video streams. This is a difficult task that requires extracting meaningful spatio-temporal features to capture activities, accurately localizing the start and end times of each activity. We introduce a new model, Region Convolutional 3D Network (R-C3D), which encodes the video streams using a three-dimensional fully convolutional network, then generates candidate temporal regions containing activities, and finally classifies selected regions into specific activities. Computation is saved due to the sharing of convolutional features between the proposal and the classification pipelines. The entire model is trained end-to-end with jointly optimized localization and classification losses. R-C3D is faster than existing methods (569 frames per second on a single Titan X Maxwell GPU) and achieves state-of-the-art results on THUMOS’14. We further demonstrate that our model is a general activity detection framework that does not rely on assumptions about particular dataset properties by evaluating our approach on ActivityNet and Charades. Our code is available at http://ai.bu.edu/r-c3d/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7f8324dda6261ec293e9705c13a0e96b9ab63474

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, background): “With more videos ﬂourishing on the internet, recognizing human actions from videos [37, 50, 9, 21, 41, 51, 52] has drawn increasing attention in computer vision community.”
Gaussian Temporal Awareness Networks for Action Localization (2019, background): “Once the localization of action proposals completes, the natural way for temporal action detection is to further classify the proposals into known action classes, making the process in two-stage manner [4, 12, 31, 32, 40, 45].”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “, action temporal localization [45, 50], video summarization [19, 30, 49] and video caption [27, 47, 53], etc) have been developed recently.”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, background): “Yuan et al. [47] 51.4 42.6 33.6 26.1 18.8 – – – – Escorcia et al. [7] – – – – 13.9 – – – – Shou et al. [27] – – 40.1 29.4 23.3 13.107.9 – – Yuan et al. [48] 51.0 45.2 36.5 27.8 17.8 – – – – Xu et al. [45] 54.5 51.5 44.8 35.6 28.9 – – – – Zhao et al. [49] 66.0 59.4 51.9 41.0 29.8 – – – – Alwasssel et”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “Here we use pre-deﬁned anchor segments for M loc , whereas for M cls we update the anchor segments by applying their predicted offsets from the M loc module (we use the same mechanism to update the segment boundaries with predicted offsets as in [41]).”
Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations (2018, background): “With more videos flourishing on the internet, videobased applications such as action recognition, action localization, and video captioning [4, 5, 33, 38, 39, 47] have drawn more and more attention.”
ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos (2020, neutral): “0.5 Glimpses [5] 48.9 44.0 36.0 26.4 17.1 - SMS [26] 51.0 45.2 36.5 27.8 17.8 - M-CNN [27] 47.7 43.5 36.3 28.7 19.0 41.2 CDC [28] - - 41.3 30.7 24.7 31.5 TURN [29] 54.0 50.9 44.1 34.9 25.6 44.8 R-C3D [30] 54.5 51.5 44.8 35.6 28.9 52.2 SSN [31] 66.0 59.4 51.9 41.0 29.8 - A-Search [12] - - 51.8 42.4 3”
SLAC: A Sparsely Labeled Dataset for Action Classification and Localization (2017, neutral): “[37] 44 .”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, background): “Besides action recognition, other tasks of action understanding, including action detection and localization [14, 47, 54, 21, 16, 36], action segmentation [26, 9], and action generation [28, 41], also attract many researchers.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “We follow Xu et al. [65]’s setting for training and testing.”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, background): “These datasets have substantially advanced the progress of research on action proposal generation [17, 18, 5, 35] and action localization [59, 48, 61, 34, 9, 2, 4].”
Temporal Action Detection with Structured Segment Networks (2017, background): “2015), R-C3D (Xu et al. 2017) proposes temporal proposals and determines its action class from a fully-convolutional 3Dnetworks.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “%) and compare with (1) P3D [47], which simplifies 3D convolutions with 2D filters on spatial dimension and 1D temporal connections; (2)R3D [67], which encodes the video sequences using a 3D fully convolutional networks and then generates candidate temporal fragments for classification; (3)I3D [3],…”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Second, proposal-based techniques [27, 31, 36, 38] extract all action proposals and refine their boundaries for action detection.”
Multilevel Language and Vision Integration for Text-to-Clip Retrieval (2018, methodology): “In this paper, we use a proposal-based pipeline to solve the video language localization task, and adopt the proposal generation technique of R-C3D.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, methodology): “[51] followed the original Faster R-CNN and thus their anchors at each pixel location still shared the receptive field.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “The first stage is to generate proposals, and then classify and refine the boundaries of these proposals in the second stage [32], [33], [34], [35], [36], [37], [38], [39], [40].”
Diagnosing Error in Temporal Action Detectors (2018, neutral): “BU [43].”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, background): “Its recent success [62,68] has evolved around a fully supervised setting, which considers the availability of frame-wise activity labels.”