Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

By Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, Kevin P. Murphy. 2017

1,502 citations recorded by the source corpus.

Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level “semantic” features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “On the other hand, it could be that FVD suffers out-of-domain and is not sufﬁciently sensitive to long-range temporal dynamics, since it is trained to perform human action recognition, which is known to predominantly rely on local features (Carreira & Zisserman, 2017; Xie et al., 2018).”
Learning Video Representations from Textual Web Supervision (2020, methodology): “In our work, we use S3D-G [67] and 3D ResNet-50 (R3D-50) [26] as the backbone 3D CNN architectures.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “For the video backbones, we observe that TSM ResNet50 always beats S3D-G for downstream tasks that involve vision.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Fully-supervised SOTA [85] S3D 75.”
Multiview Transformers for Video Recognition (2022, neutral): “Since the release of the Kinetics dataset [33], 3D CNNs [10, 24, 69] have gained popularity, and many variants [22, 63, 70, 71, 79] have been developed to improve the speed and accuracy.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “For a long time this space was dominated by spatio-temporal CNNs such as I3D [8], 3D ResNets [18], S3D [64] or ‘R(2+1)D’ CNNs [55].”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “We use a frozen S3D [85] pretrained on HowTo100M [60] using MIL-NCE [59].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “, XYT) convolutional layers as well as (2+1)D convolutional layers have been popularly used to represent videos (Tran et al., 2014; Carreira & Zisserman, 2017; Tran et al., 2018; Xie et al., 2018).”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “In the case when there are multiple frames per step, we combine the image network with an S3D (Xie et al., 2018) inspired network, which adds additional temporal convolutions on spatial feature maps.”
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “9% on Kinetics [40] and 88% on ActivityNet [35].”
Is Space-Time Attention All You Need for Video Understanding? (2021, neutral): “We note that whereas most previous Method Top-1 Top-5 TFLOPs R(2+1)D (Tran et al., 2018) 72.0 90.0 17.5 bLVNet (Fan et al., 2019) 73.5 91.2 0.84 TSM (Lin et al., 2019) 74.7 N/A N/A S3D-G (Xie et al., 2018) 74.7 93.4 N/A Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84 D3D (Stroud et al., 2020) 75.9 N/A ”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “[53] proposed to study mixed 2D and 3D networks, either first using 3D and later 2D (bottom-heavy) or first 2D and later 3D (top-heavy) architecture.”
ViViT: A Video Vision Transformer (2021, background): “As 3D convolutional networks require signiﬁcantly more computation than their image counterparts, many architectures factorise convolutions across spatial and temporal dimensions and/or use grouped convo-lutions [59, 66, 67, 81, 20].”