A discriminative CNN video representation for event detection

By Zhongwen Xu, Yi Yang, Alexander G. Hauptmann. 2014

462 citations recorded by the source corpus.

In this paper, we propose a discriminative video representation for event detection over a large scale video dataset when only limited hardware resources are available. The focus of this paper is to effectively leverage deep Convolutional Neural Networks (CNNs) to advance event detection, where only frame level static descriptors can be extracted by the existing CNN toolkits. This paper makes two contributions to the inference of CNN video representation. First, while average pooling and max pooling have long been the standard approaches to aggregating frame level static features, we show that performance can be significantly improved by taking advantage of an appropriate encoding method. Second, we propose using a set of latent concept descriptors as the frame descriptor, which enriches visual information while keeping it computationally affordable. The integration of the two contributions results in a new state-of-the-art performance in event detection over the largest video datasets. Compared to improved Dense Trajectories, which has been recognized as the best video representation for event detection, our new representation improves the Mean Average Precision (mAP) from 27.6% to 36.8% for the TRECVID MEDTest 14 dataset and from 34.0% to 44.6% for the TRECVID MEDTest 13 dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “[41] propose to utilize VLAD [12] to aggregate frame level ConvNet for video representation, which is unable to capture temporal structure.”
Actions ~ Transformations (2015, background): “Learned representations with deep learning have recently produced state of the art results in action recognition [17, 21, 39, 52, 48, 49, 28, 8, 58].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, background): “Many attempts in this genre leverage CNNs trained on images to extract features from the individual frames and then perform temporal integration of such features into a fixed-size descriptor using pooling, high-dimensional feature encoding [41, 11], or recurrent neural networks [42, 7, 32, 2].”
HCP: A Flexible CNN Framework for Multi-Label Image Classification (2014, neutral): “, ImageNet [10]) and event detection [42].”
Asynchronous Temporal Fields for Action Recognition (2016, background): “With the advent of deep learning, learning representations from data has been extensively studied [14, 15, 44, 57, 52, 53, 24, 7, 61, 55, 40, 3].”
Learnable pooling with Context Gating for video classification (2017, methodology): “Application of these techniques to video include [7], [8], [9], [34], [35].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, methodology): “VLAD [15] and NetVLAD [1] have achieved great impacts in aggregating discriminative features for video classification [10, 36], video retrieval [24], person re-identification [40].”
Encoding Feature Maps of CNNs for Action Recognition (2015, unknown): “In [9], VLAD and Fisher vector are applied to fc6 activations and pool5 feature maps for event detection on TRECVID MED dataset.”
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (2019, background): “From traditional hand-crafted features [32, 18, 4, 7, 31, 41] to deep neural networks derived features [44, 11, 8, 35, 15, 23], the improvements are obvious.”
Recognizing and Curating Photo Albums via Event-Specific Image Importance (2017, methodology): “The most popular approach uses videos as input [7, 17, 18, 22, 26, 29, 30], and spatiotemporal features are commonly used.”
The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection (2016, methodology): “The leading approaches [7, 12, 15, 25] attack this challenging problem by learning video representations through a deep convolutional neural network [9, 22].”
SST: Single-Stream Temporal Action Proposals (2017, unknown): “[36, 9].”
Bidirectional Multirate Reconstruction for Temporal Modeling in Videos (2016, methodology): “[49] applied VLAD encoding [18] over ConvNet activations and found that the encoding methods are superior to mean pooling.”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, unknown): “Some works proposed encoding methods [18,19,20] to learn a video representation from samples.”
Semantic Pooling for Complex Event Analysis in Untrimmed Videos (2016, background): “3 kernels are more suitable for spatiotemporal features such as in human action recognition; [25] achieved very impressive performance improvements on event detection by incorporating a set of latent concept descriptors, appropriately encoded using the vector of locally aggregated descriptors…”
Watching a Small Portion could be as Good as Watching All: Towards Efficient Video Classification (2018, background): “…efforts have been made to improve video classiﬁcation performance in recent years [Wang et al. , 2011; Simonyan and Zisserman, 2014; Karpathy et al. , 2014; Wang et al. , 2016; Tran et al. , 2015; Ng et al. , 2015; Chang et al. , 2016; Fan et al. , 2017; Xu et al. , 2015; Bhat-nagar et al. , 2017].”
The THUMOS challenge on action recognition for videos "in the wild" (2016, unknown): “Based on the results, we make the following observations: 1) The LCD coding with the VLAD representation ( Xu et al., 2015a ) is very effective; 2) ﬁne-tuning the CNN models can bring further improvements; and 3) some specially designed network structures for video analysis are helpful, e.g., the…”