Spatiotemporal Residual Networks for Video Action Recognition

By Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes. Neural Information Processing Systems, 2016

745 citations recorded by the source corpus.

Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping these with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://arxiv.org/abs/1611.02155

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Non-local Neural Networks (2017, methodology): “As done in [13, 7], one can turn the C2D model in Table 1 into a 3D convolutional counterpart by “inflating” the kernels.”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “[28] and [9] generalized the residual networks (ResNets) for the spatiotemporal domain for video action recognition and”
Multiview Transformers for Video Recognition (2022, neutral): “Inspired by ViT, ViViT [3] and Timesformer [6] were the ﬁrst two works that successfully adopted a pure transformer architecture for video classiﬁcation, advancing the state of the art previously set by 3D CNNs. Multiscale processing in computer vision .”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, methodology): “7 on HMDB-51, which correspond to 63% and 35% misclassification reductions, respectively compared to the best previous model [7].”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
SlowFast Networks for Video Recognition (2018, methodology): “, [12, 49, 5, 56]) that works on a clip of video as a spatiotemporal volume.”
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, methodology): “Many methods based on the two-stream CNNs are proposed to improve action recognition performance [5, 6, 20, 22].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Using two-stream network designs has also been extremely popular in video CNNs [20, 7–9].”
Video Representation Learning Using Discriminative Pooling (2018, background): “Comparison to standard pooling on HMDB-51 split-1 VGG ResNet Spatial Stream-AP[10, 13] 47.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “, 2016), Feichtenhofer et al. (2016a) injected residual connections between the”
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation (2018, methodology): “One possibility is to use i3D [44] or non-local neural network from Wang et al. [45].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, methodology): “We train our R(2+1)D architecture on both RGB and optical flow inputs and fuse the prediction scores by averaging, as proposed in the two-stream framework [29] and subsequent work [4, 9].”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Using the 2D CNN is a straightforward way to conduct video recognition [24, 39, 48, 11, 8, 9, 2].”
ViViT: A Video Vision Transformer (2021, methodology): “…ﬁlters from 2D ﬁlters for video classiﬁcation is to “inﬂate” them by replicating the ﬁlters along the temporal dimension and averaging them [8, 22] as We consider an additional strategy, which we denote as “central frame initialisation”, where E is initialised with ze-roes along all temporal…”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, background): “This has inspired a new generation of deep video models [17, 18, 19] greatly advancing the field.”
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, methodology): “Since that study, numerous methods based on the two-stream CNNs have been proposed to improve action recognition performance [6, 7, 8, 27, 28, 29].”
Temporal Segment Networks for Action Recognition in Videos (2017, neutral): “VGG-M [69] (from [1]) 73:0% 83:7% 86:9% GoogLeNet [9] (from [70]) 75:3% 85:8% 89:3% VGGNet-16 [8] (from [70]) 78:4% 87:0% 91:4% ResNet-152 [31] (from [71]) 83:4% 87:2% 91:8% BN-Inception [62] 85:0% 88:3% 92:4%”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “Another strategy is to use aggressive subsampling [50, 58] or large striding [8].”