Convolutional Two-Stream Network Fusion for Video Action Recognition

By Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman. 2016

2,778 citations recorded by the source corpus.

Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters, (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy, finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “As a method, the two-stream fusion architecture in [8] creates spatio-temporal relationship with extra 3D convolution layer inserted towards the end of the architecture and fuses information from RGB and optical flow streams.”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, methodology): “Among 2D ConvNet meth-ods, we consider ConvNets with LSTMs on top [5, 34], and two-stream networks with two different types of stream fusion [8, 25].”
The Kinetics Human Action Video Dataset (2017, methodology): “There have been many improvements over these basic architectures, e.g. [9], but our intention here is not to perform a thorough study on what is the very best architecture on Ki-netics, but instead to provide an indication of the level of difﬁculty of the dataset.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
SlowFast Networks for Video Recognition (2018, methodology): “This method has been a foundation of many competitive results in the literature [12, 13, 55].”
Spatiotemporal Residual Networks for Video Action Recognition (2016, methodology): “…are created by weighted averaging of video frames over time; [31] captures the transformation of ConvNet features from the beginning to the end of the video with a Siamese architecture; and [5] introduces a spatiotemporal convolutional fusion layer between the streams of a two-stream architecture.”
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, methodology): “Many methods based on the two-stream CNNs are proposed to improve action recognition performance [5, 6, 20, 22].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, positive): “Using two-stream network designs has also been extremely popular in video CNNs [20, 7–9].”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “For example, [10] studies how to fuse the information from two different modalities, RGB and motion, using 2D CNNs. Wang et al .”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “Following the recent trends, for this experiment, we use a two-stream CNN model in popular architectures, the VGG-16 and the ResNet-152 [13, 42].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[32] investigated three aspects of fusion for the two streams: (i) how to fuse the two networks with consideration for spatial registration, (ii) where to fuse the two networks and, (iii) how to fuse the networks temporally.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, methodology): “[9] investigated the optimal position within a convolution network in detail to combine the separate streams.”
Long-Term Temporal Convolutions for Action Recognition (2016, neutral): “these datasets, except [25] which studies best ways to combine RGB and flow streams, hence complementary to our method.”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, background): “The idea of combining two features using the outer product has also been shown to be effective for other tasks such as visual question-answering [16] where text and visual features are combined, action recognition [14] where ﬂow and image features are combined.”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, methodology): “Additional two-stream approaches include Temporal Segment Networks [35], Action Transformations [36], and Convolutional Fusion [10].”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Most prior works [12], [31], [33], [37], [38] address the problem of appearance and motion fusion for video representation.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Using the 2D CNN is a straightforward way to conduct video recognition [24, 39, 48, 11, 8, 9, 2].”
Learning and Using the Arrow of Time (2018, methodology): “[5] to give an extended temporal footprint; and the CAM model of Zhou et al.”
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, methodology): “Since that study, numerous methods based on the two-stream CNNs have been proposed to improve action recognition performance [6, 7, 8, 27, 28, 29].”