A Robust and Efficient Video Representation for Action Recognition

By Heng Wang, Dan Oneaţă, Jakob Verbeek, Cordelia Schmid. International Journal of Computer Vision, 2015

339 citations recorded by the source corpus.

This paper introduces a state-of-the-art video representation and applies it to efficient action recognition and detection. We first propose to improve the popular dense trajectory features by explicit camera motion estimation. More specifically, we extract feature point matches between frames using SURF descriptors and dense optical flow. The matches are used to estimate a homography with RANSAC. To improve the robustness of homography estimation, a human detector is employed to remove outlier matches from the human body as human motion is not constrained by the camera. Trajectories consistent with the homography are considered as due to camera motion, and thus removed. We also use the homography to cancel out camera motion from the optical flow. This results in significant improvement on motion-based HOF and MBH descriptors. We further explore the recent Fisher vector as an alternative feature encoding approach to the standard bag-of-words (BOW) histogram, and consider different ways to include spatial layout information in these encodings. We present a large and varied set of evaluations, considering (i) classification of short basic actions on six datasets, (ii) localization of such actions in feature-length movies, and (iii) large-scale recognition of complex events. We find that our improved trajectory features significantly outperform previous dense trajectories, and that Fisher vectors are superior to BOW encodings for video recognition tasks. In all three tasks, we show substantial improvements over the state-of-the-art results.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bfb86d1c931af850a16a597108ba44a20bda8f57

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Human Action Recognition and Prediction: A Survey (2018, methodology): “The action representation component basically converts an action video into a feature vector [11], [80], [81], [82] or a series of vectors [21], [54], [83], and the action classification component infers an action label from the vector [53], [84], [85].”
AMTnet: Action-Micro-Tube Regression by End-to-end Trainable Deep Architecture (2017, background): “While many works concern either spatial action localisation [18, 32, 10, 25] in trimmed videos or temporal localisation [17, 3, 27, 19, 31, 23, 35, 34] in untrimmed videos, only a handful number of methods have been proposed to tackle both problems jointly.”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, background): “Since then, bigger and more realistic datasets have been proposed [37, 43, 19, 41], and hand-crafted features have also improved significantly [49, 44, 48].”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “The iDT [62] method is often recognized as the best performing method with handengineered features [63].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[93] removed the background trajectories, and warped optical flow with an estimated homography approximating the camera motion to improve the accuracy.”
A Comprehensive Survey of Vision-Based Human Action Recognition Methods (2019, background): “[63] fused the results of human detection into iDT features, thereby reducing the interference of background trajectories.”
Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks (2016, methodology): “We compare our approach to state-of-theart methods: HOG+HOG2 [27], improved dense trajectories (iDT) [39], super normal vector (SNV) [42], two-stream CNNs [34], and convolutional 3D (C3D) [37], as well as human labeling accuracy.”
Encoding Feature Maps of CNNs for Action Recognition (2015, background): “Improved dense trajectories (IDT) based video representations have shown excellent performance on many action datasets [7].”
Online Real-Time Multiple Spatiotemporal Action Localisation and Prediction (2016, methodology): “Temporal action detection is mostly tackled using expensive sliding window [20, 5, 45, 26, 50] approaches.”
Learning to Track for Spatio-Temporal Action Localization (2015, positive): “[44] achieve state-of-the-art performance using Fisher Vectors and dense trajectories with motion stabilization.”
AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos (2016, methodology): “We extract the iDT features using the executables provided by the authors [40] and use human bounding boxes for HMDB51 but not for UCF101.”
Human Action Localization with Sparse Spatial Supervision (2017, methodology): “In this paper we rely on two detectors, one based on dense trajectories [1] and the other on CNNs [19].”
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition (2016, methodology): “A signiﬁcant part of the progress on action recognition is driven by the development of local hand-crafted spatio-temporal features encoded as bag-of-words representations classiﬁed by “shallow” classiﬁers such as SVMs [2,3,4,5,6,7,8,9,10].”
Hierarchical motion evolution for action recognition (2015, background): “Significant progress has been made in recent years by designing local spatio-temporal features and adopting different encoding schemes [16, 24].”
Towards Weakly-Supervised Action Localization (2016, methodology): “Action classification has been widely studied over the past decade and state-ofthe-art methods [1,2,3,4] now achieve excellent performance.”