Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

By Zhaofan Qiu, Ting Yao, Tao Mei. IEEE International Conference on Computer Vision, 2017

1,834 citations recorded by the source corpus.

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating 3 x 3 x 3 convolutions with 1 × 3 × 3 convolutional filters on spatial domain (equivalent to 2D CNN) plus 3 × 1 × 1 convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/024d037d46ae933c7e12fd16af61953c7161773a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “69 C3D [33] Sports1M 65.”
Spatio-Temporal Channel Correlation Networks for Action Classification (2018, neutral): “…HMDB51 DT+MVSM [34] 83.5 55.9 iDT+FV [35] 85.9 57.2 C3D [2] 82.3 56.8 Conv Fusion [17] 82.6 56.8 Two Stream [16] 88.6 − TDD+FV [36] 90.3 63.2 TSN-RGB [37] 85.7 − P3D [38] 88. fully connected layers (C3D w/ BN), and RGB-I3D which is without pretraining on the ImageNet (RGB-I3D w/o ImageNet) [19].”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, methodology): “Therefore, we choose the pseudo-3D convolution proposed in [28] that decomposes 3D learning into 2D convolutions in spatial space and 1D operations in temporal dimension.”
TransMed: Transformers Advance Multi-Modal Medical Image Classification (2021, methodology): “The input-level fusion strategy and the decision-level fusion strategy can be implemented using mainstream 2D CNN and 3D CNN, so the selected network includes Resnet34, Resnet152, 3D Resnet34, P3D [49], C3D [50], and BoTNet50 [31].”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, background): “A majority of modern approaches have introduced deep architecture models [10, 59, 57, 62], including the most prevailing two-stream networks [57], C3D [59] and their variants [62, 15, 53, 10].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Most current approaches [50, 30, 56, 9, 45, 57, 65, 15] first train a clipclassifier Ω(·) to operate on a short fixed-length video clip V ∈ RF×3×H×W of F frames with spatial resolution H × W , typically spanning several seconds.”
Spatio-Temporal Graph Structure Learning for Traffic Forecasting (2020, methodology): “By modifying C3D (Tran et al. 2015), P3D is not only an economic and effective way for spatio-temporal featuring, but also an efﬁcient way for computing (Qiu, Yao, and Mei 2017).”
Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition (2018, positive): “CNN+RNN architecture for video sequence modelling [2, 3] and purely ConvNet-based video recognition [4, 5, 6, 7, 8, 1, 9] are two major research directions.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “They extend 2D image models with a temporal dimension, most notably incorporating 3D convolution [28, 62, 63, 72, 23, 49, 29, 52].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, methodology): “Other works separately factor convolutions into separate 2D spatial and 1D temporal filters [25, 61, 66, 49].”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Tran et al. (Tran et al. 2019) and Qiu et al. (Qiu, Yao, and Mei 2017) propose to decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution.”
ModelScope Text-to-Video Technical Report (2023, methodology): “To ensure effectively synthesising videos, we factorise the convolution and the attention mechanism over space and time [41, 1].”
Lower Dimensional Kernels for Video Discriminators (2019, positive): “…supported by results from related domains such as video recognition, where (Feichtenhofer, Pinz, & Wildes, 2016; Lin, Gan, & Han, 2018; Qiu, Yao, & Mei, 2017; Sun, Jia, Yeung, & Shi, 2015; Tran et al., 2018; Xie, Sun, Huang, Tu, & Murphy, 2018) have successfully removed 3D kernels from…”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, neutral): “Other works separately control spatial and temporal dimensions factoring convolutions into separate 2D spatial and 1D temporal filters [29, 93, 98, 72].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “With the success of applying 3D convolution on video analysis tasks, various 3DConvNet architectures have been proposed [69], [70], [71].”
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (2019, methodology): “The procedure can be refactored by first applying spatial convolution then temporal convolution [30, 36].”
Low-Memory Neural Network Training: A Technical Report (2019, background): “For certain datasets, such as video (Hara et al., 2017) and high-resolution medical images (Qiu et al., 2017), the prohibitive memory requirements of current training algorithms often mean that smaller, less expressive models must be used or the inputs must be downsampled to a lower dimensionality,…”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, methodology): “There have already existed pretty much work on improving the efficiency of action recognition in videos [12, 35, 20, 2, 34, 27, 22, 28, 25, 19].”