Beyond short snippets: Deep networks for video classification

By Joe Yue-Hei Ng, Matthew J. Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici. 2015

2,456 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Semi-supervised Sequence Learning (2015, methodology): “We also experiment with the approach of putting the label at every timestep and linearly increasing the weights of the prediction objec tiv s from 0 to 1 [24].”
Non-local Neural Networks (2017, methodology): “A natural solution to video classification is to combine the success of CNNs for images and RNNs for sequences [55, 11].”
Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “Pooling is a well-known technique to combine various temporal features; concatenation, averaging, maximum, minimum, ROI, feature aggregation techniques and time-domain convolution are some of the possible pooling techniques [11,21].”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, background): “video action classification [13, 43], image captioning [40, 21] or visual question answering [2]).”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “, [3], [4], [5], [6], [7], [8], [9], and [10], however, the progress in developing generative models of dynamic patterns for synthesis purpose has been lagging behind.”
Multiview Transformers for Video Recognition (2022, background): “At the same time, CNNs and RNNs have quickly become the de-facto backbones for video understanding tasks [32, 50, 57].”
Long-term recurrent convolutional networks for visual recognition and description (2014, neutral): “Contemporaneous with our work, [66] train a network which combines CNNs and LSTMs for activity recognition.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classification methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
Sequence to Sequence -- Video to Text (2015, methodology): “Others [24, 8] have shown that incorporating optical flow information to LSTMs improves activity classification.”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, methodology): “0 LSTM with Optical Flow and Pixels [26] - 73.”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, methodology): “for encoding spatial dependencies in segmentation [53] and video classification [30].”
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition (2016, background): “Recently, several works focused on modeling long-range temporal structure with ConvNets [4,19,20].”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “RNN models have also been used to encode temporal information for learning video representations in conjonction with 2D CNNs. Ng et al. (2015); Donahue et al. (2014) applied an RNN on top of the the two-stream framework, while Srivastava et al. (2015) proposed, in addition, to investigate the…”
Spatiotemporal Residual Networks for Video Action Recognition (2016, methodology): “For both datasets, we use the provided evaluation protocol and report mean average accuracy over three splits into training and test sets.”
Self-Driving Car Steering Angle Prediction Based on Image Recognition (2019, neutral): “[20].”
Peephole: Predicting Network Performance Before Training (2017, background): “in language modeling [19] and video analytics [25], we choose to explore recurrent networks in our problem.”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “[84] studied two techniques of feeding videos to convnets for video classification.”
Video Representation Learning Using Discriminative Pooling (2018, background): “Several works try to tackle this problem by using different pooling strategies [2, 14, 53, 54, 60], which achieve some improvement compared with the baseline algorithm.”