A Closer Look at Spatiotemporal Convolutions for Action Recognition

By Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, Manohar Paluri. 2017

3,656 citations recorded by the source corpus.

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly gains in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which produces CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101, and HMDB51.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “R(2+1)D [28] Kinetics-400 74.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Using the same visual backbone (R(2+1)D-18 [82]), training dataset (AudioSet) and modalities (Vision+Audio), we obtain similar performance to XDC [7] and GDT [65] on UCF101, and significantly outperform them on HMDB51.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “Then, we train a ResNet-3D [41] network with a softmax classi-ﬁcation loss.”
Billion-scale semi-supervised learning for image classification (2019, methodology): “We use R(2+1)D-d [40] clip-based models with model depths 18 and 34 and use RGB video frames.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “This idea has been used in a variety of recent papers, including [11] (who call it “R(2+1)D”), [15] (who call it “Pseudo-3D network”), [16] (who call it “factorized spatio-temporal convolutional networks”), etc.”
SlowFast Networks for Video Recognition (2018, methodology): “There are also related methods focusing on long-term filtering and pooling using temporal strides [52, 13, 55, 62], as well as decomposing the convolutions into separate 2D spatial and 1D temporal filters [12, 50, 61, 39].”
Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans (2020, methodology): “the 3D models, we use DenseNet3D121 [17], R2Plus1D [18], MC3 18 [18], ResNeXt3D101 [17], PreAct ResNet3D [17], and ResNet3D series [17] (ResNet3D10, ResNet3D18, ResNet3D34, ResNet3D50, ResNet3D101, ResNet3D152, and ResNet3D200).”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Full 3D space-time (i.e., XYT) convolutional layers as well as (2+1)D convolutional layers have been popularly used to represent videos [22, 3, 23, 30].”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “Recent approaches handle this issue in diverse ways, for example, by applying group convolutions [37], learning 3D shift operations [7], decomposing 3D convolution ﬁlters [38], searching efﬁcient architectures [8], etc .”
Is Space-Time Attention All You Need for Video Understanding? (2021, neutral): “We note that whereas most previous Method Top-1 Top-5 TFLOPs R(2+1)D (Tran et al., 2018) 72.0 90.0 17.5 bLVNet (Fan et al., 2019) 73.5 91.2 0.84 TSM (Lin et al., 2019) 74.7 N/A N/A S3D-G (Xie et al., 2018) 74.7 93.4 N/A Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84 D3D (Stroud et al., 2020) 75.9 N/A…”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Another way to save computation is to decompose the 3D convolution into a 2D spatial convolution and a 1D temporal convolution [46, 33, 42].”
Video Action Transformer Network (2018, neutral): “…[7], S3D(+G): Separable 3D convolutions (with gating) [45], FRCNN: Faster R-CNN [31], NL: Non-local networks [44], P3D: Pseudo-3D convo-lutions [30], C2D [40], C3D [40], TSN: Temporal Segment Networks [43] RN: Relation Nets [32], Tx: Transformer [29, 41] and FPN: Feature Pyramid Networks [26].”
ViViT: A Video Vision Transformer (2021, background): “As 3D convolutional networks require signiﬁcantly more computation than their image counterparts, many architectures factorise convolutions across spatial and temporal dimensions and/or use grouped convo-lutions [59, 66, 67, 81, 20].”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “While in RNNs, appearance and motion are modeled separately, 3D and 2D/3D hybrid convolutional operators [34, 28] intrinsically integrates spatio-temporal visual information and are also used for VideoQA [10, 20].”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “The video subnetwork (shown in (b)) is a MCx network [19] using 3D convolutions in the early layers, and 2D convolutions in the subsequent layers.”