Two-Stream Convolutional Networks for Action Recognition in Videos

By Karen Simonyan, Andrew Zisserman. 2014

8,273 citations recorded by the source corpus.

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking ImageNet Pre-Training (2018, background): “Pre-training has enabled state-of-the-art results on many tasks, including object detection [9, 8, 36], image segmentation [29, 13], and action recognition [42, 4].”
What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
Relation Networks for Object Detection (2017, background): “We do not see any reason preventing it from finding broader applications in vision tasks, such as instance segmentation [30], action recognition [41], object relationship detection [28], caption [50], VQA [1], etc.”
Non-local Neural Networks (2017, background): “In addition to endto-end modeling on raw video inputs, it has been found that optical flow [45] and trajectories [50, 51] can be helpful.”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, background): “These visual representations have been shown to be effective on a wide range of computer vision tasks [1, 4, 14, 22, 29, 33, 36].”
Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, neutral): “Two-Stream [24] X ImageNet 59.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Most existing methods [23, 34, 38, 39] rely on supervised learning and, therefore, require large-scale, diverse and representative action datasets for training.”
Contrastive Multiview Coding (2019, background): “The spatial network from [69] is a well-established paradigm for evaluating pre-trained RGB network on action recognition task.”
Spatial Transformer Networks (2015, background): “cent invention, we now see a cornucopia of CNN-based models achieving state-of-the-art results in classiﬁcation [19, 28, 35], localisation [31, 37], semantic segmentation [24], and action recognition [12, 32] tasks, amongst others. A desirable property of a system which is able to reason about image”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, background): “tection [29, 23], semantic segmentation [1, 24], and video analysis [27, 31].”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (2016, neutral): “Visual tasks, such as object classification and detection, have been successfully solved through the supervised learning paradigm [1,10,22,31], where one uses labeled data to train a parametric model.”
Hypercolumns for object segmentation and fine-grained localization (2014, background): “Features based on convolutional networks (CNNs) [29] have now led to the best results on a range of vision tasks: image classification [28, 36], object segmentation and detection [18, 22], action classification [35], pose estimation [37] and fine-grained category recognition [44, 6].”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “Carreira and Zisserman [10] proposed the Inception 3D (I3D) model by inflating all the 2D convolution filters and pooling kernels used by the Inception V1 architecture [29] into 3D convolutions and pre-training the model on the large-scale Kinetics human action video dataset [30].”
Joint Discovery of Object States and Manipulation Actions (2017, background): “Most of the prior work on action recognition has focused on designing features to describe time intervals of a video using motion and appearance [30, 39, 43, 44].”
Generating Videos with Scene Dynamics (2016, methodology): “We also use two-streams to model video [34], but apply them for video generation instead of action recognition.”
Deep multi-scale video prediction beyond mean square error (2015, neutral): “…algorithms for action recognition exploit the temporal information in a supervised way, such as the 3D convolutional network of Tran et al. (2015), or the spatio-temporal convolutional model of Simonyan & Zisserman (2014), which can require months of training, and heavily labeled datasets.”
Multiview Transformers for Video Recognition (2022, background): “At the same time, CNNs and RNNs have quickly become the de-facto backbones for video understanding tasks [32, 50, 57].”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Language-guided visual reasoning like VQA has progressed significantly driven by the tremendous advancements in object/action recognition [4, 13, 14, 40, 44] and natural language understanding [5, 7, 15, 35, 45].”