Exploiting Image-trained CNN Architectures for Unconstrained Video Classification

By Shengxin Zha, Florian Luisier, Walter Andrews, Nitish Srivastava, Ruslan Salakhutdinov. 2015

209 citations recorded by the source corpus.

We conduct an in-depth exploration of different strategies for doing event detection in videos using convolutional neural networks (CNNs) trained for image classification. We study different ways of performing spatial and temporal pooling, feature normalization, choice of CNN layers as well as choice of classifiers. Making judicious choices along these dimensions led to a very significant increase in performance over more naive approaches that have been used till now. We evaluate our approach on the challenging TRECVID MED'14 dataset with two popular CNN architectures pretrained on ImageNet. On this MED'14 dataset, our methods, based entirely on image-trained CNN features, can outperform several state-of-the-art non-CNN models. Our proposed late fusion of CNN- and motion-based features can further increase the mean average precision (mAP) on MED'14 from 34.95% to 38.74%. The fusion approach achieves the state-of-the-art classification performance on the challenging UCF-101 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Action Recognition using Visual Attention (2015, methodology): “We describe how our model dynamically pools convolutional features and show that using these features for action recognition gives better results compared to average or max pooling which is used by many of the existing models (Zha et al., 2015).”
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos (2015, methodology): “[40] examine aggregation strategies for combining deep learned image-based features for each frame, obtaining impressive results on TRECVID MED retrieval.”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, neutral): “This is consistent with other work on CNNs for action recognition using max-pooling [48], highlighting the importance of learning effective frame observation policies.”
Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification (2015, methodology): “On the UCF-101, many works with competitive results are based on the dense trajectories [44, 52], while our approach fully relies on the deep learning techniques.”
Anticipating the future by watching unlabeled video (2015, methodology): “Moreover, these representations have been empirically shown to work on a large variety of tasks [33, 5, 50].”
Bidirectional Multirate Reconstruction for Temporal Modeling in Videos (2016, unknown): “VGG + RBF [53] 35.”
Regularizing Long Short Term Memory with 3D Human-Skeleton Sequences for Action Recognition (2016, background): “However, despite the ongoing research efforts to: (a) Increase the amount of training data [15], (b) Fuse hand-designed and deep-convolutional features [35, 26, 41], and (c) Combine CNNs with either graphical models [31, 11, 39], or recurrent neural networks [4, 39] for capturing complex dynamics of”
Combining multiple sources of knowledge in deep CNNs for action recognition (2016, methodology): “Another straightforward way to improve performance would be to combine various hand-crafted features with CNN features, which might also be helpful for our method[28].”
Action Recognition by Learning Deep Multi-Granular Spatio-Temporal Video Representation (2016, methodology): “Please note that the methods in [29, 18, 13] are based on traditional dense trajectory which is computationally expensive, while the methods in [35, 30] combine dense trajectory and deep learning based algorithms.”
Learning Temporal Embeddings for Complex Video Analysis (2015, methodology): “Additionally, the Imagenet pre-trained CNN features seem to perform better than most previous representations, which is also consistent with previous work [41, 44].”
UvA-DARE (Digital Academic Repository) Dynamic Image Networks for Action Recognition (2016, methodology): “Considering deep learning methods, our method performs on par and is only outperformed from [33].”
Improved Dense Trajectory with Cross Streams (2016, unknown): “Figure 1: Illustration of visualized iDT and feature map from convolutional layer in temporal net.”
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition (2016, methodology): “[34] combine CNN features trained on ImageNet [43] with iDT features through a Kernel SVM.”
Learning End-to-end Video Classification with Rank-Pooling (2016, methodology): “Recently, several methods have been proposed to tackle the problem of video sequence encoding using neural network architectures (Ji et al., 2013; Donahue et al., 2015; Srivastava et al., 2015; Yue-Hei Ng et al., 2015; Zha et al., 2015; Simonyan & Zisserman, 2014).”
BING: Binarized normed gradients for objectness estimation at 300fps (2014, unknown): “In comparison, BING is based on lowlevel cues concerning enclosing boundaries and thus can produce category independent object proposals, which has demonstrated applications in multi-label image classification [23], semantic segmentation [25], video classification [24], co-salient object detection [”
Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video (2016, methodology): “Current methods explore both high-level representations based on objects, attributes, or scenes [3], [4], [8], [22], [23], as well as holistic frame-level CNN descriptors [4]–[7].”
Webly-Supervised Video Recognition by Mutually Voting for Relevant Web Images and Web Video Frames (2016, unknown): “Instead of learning representation using video data, recent works [47] for complex event recognition have shown CNN features from models pre-trained from ImageNet [5] achieve promising results.”
Semantic Pooling for Complex Event Analysis in Untrimmed Videos (2016, methodology): “, [26], that large improvements on detection accuracy can be achieved by pooling the feature representations carefully and/or using statistically more powerful classifiers.”