Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice

By Xiaojiang Peng, Limin Wang, Xingxing Wang, Yu Qiao. 2014

616 citations recorded by the source corpus.

A Comprehensive study on the BoVW pipeline for action recognition task.An evaluation and a generic analysis on 13 encoding methods.Intra-normalization for supervector based encoding methods.An evaluation on three fusion methods.Several good practices of the BoVW pipeline for action recognition task. Video based action recognition is one of the important and challenging problems in computer vision research. Bag of visual words model (BoVW) with local features has been very popular for a long time and obtained the state-of-the-art performance on several realistic datasets, such as the HMDB51, UCF50, and UCF101. BoVW is a general pipeline to construct a global representation from local features, which is mainly composed of five steps; (i) feature extraction, (ii) feature pre-processing, (iii) codebook generation, (iv) feature encoding, and (v) pooling and normalization. Although many efforts have been made in each step independently in different scenarios, their effects on action recognition are still unknown. Meanwhile, video data exhibits different views of visual patterns , such as static appearance and motion dynamics. Multiple descriptors are usually extracted to represent these different views. Fusing these descriptors is crucial for boosting the final performance of an action recognition system. This paper aims to provide a comprehensive study of all steps in BoVW and different fusion methods, and uncover some good practices to produce a state-of-the-art action recognition system. Specifically, we explore two kinds of local features, ten kinds of encoding methods, eight kinds of pooling and normalization strategies, and three kinds of fusion methods. We conclude that every step is crucial for contributing to the final recognition rate and improper choice in one of the steps may counteract the performance improvement of other steps. Furthermore, based on our comprehensive study, we propose a simple yet effective representation, called hybrid supervector, by exploring the complementarity of different BoVW frameworks with improved dense trajectories. Using this representation, we obtain impressive results on the three challenging datasets; HMDB51 (61.9%), UCF50 (92.3%), and UCF101 (87.9%).

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “Two kinds of motion representations can be identified: Lagrangian motion [48, 18, 31, 50, 30, 56] and Eulerian motion [2, 25, 60, 51, 52, 1].”
Action recognition with trajectory-pooled deep-convolutional descriptors (2015, background): “For deep-learned features, Convolutional Networks (ConvNets) [18] are popular deep architectures, which contain a sequence of convolutional and pooling layers.”
Beyond Gaussian Pyramid: Multi-skip Feature Stacking for action recognition (2014, methodology): “[24] improved the performance of Improved Dense Trajectory by increasing the codebook size and fusing multiple coding methods.”
Long-Term Temporal Convolutions for Action Recognition (2016, neutral): “ained on Sports-1M. For HMDB51, ow is pre-trained on UCF101 and RGB scores are obtained using C3D feature extractor. Long-term Temporal Convolutions 13 Method UCF101 HMDB51 [5] IDT+FV 85.9 57.2 Hand- [29] IDT+HSV 87.9 61.1 crafted [30] IDT+MIFS 89.1 65.1 [31] IDT+SFV - 66.8 [12] Slow fusion (from sc”
Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks (2015, neutral): “2% IDT higher-dimensional encodings [23] 87.”
Temporal Segment Networks for Action Recognition in Videos (2017, neutral): “iDT+HSV [75] 61:1% iDT+HSV [75] 87:9% FLOW-I3D [56] 65:3%/86:7%”
Hierarchical Coding Vectors for Scene Level Land-Use Classification (2016, methodology): “To retain spatial information, the dense local descriptors (e.g., Scale Invariant Feature Transformation (SIFT) [8]) [12], LCC sparsity caused where distance ˆ e is of likelihood to the distance ˆ e and is critical to the coding and classiﬁcation performance.”
Two-Stream Convolutional Networks for Action Recognition in Videos (2014, methodology): “deep architecture significantly outperforms that of [14] and is competitive with the state of the art shallow representations [20, 21, 26] in spite of being trained on relatively small datasets.”
Real-Time Action Recognition with Enhanced Motion Vector CNNs (2016, methodology): “ets and worldwide competitions, like UCF101 [25], HMDB51 [15], and THUMOS14 [11] promote researches in this area. Early approaches in this area utilize a Bag-of-Visual-Words paradigm and its variants [20], which mainly consists of feature extraction, feature encoding, and classiﬁcation steps. The pe”
A Review on Human Activity Recognition Using Vision-Based Method (2017, methodology): “Local features are then normally combined with the bag-of-visual-words (BoVW) model and yield the general pipeline of current state-of-the-art local representation approaches [31].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “For vocabulary building, one of the popular methods is the Bag-of-words (BoW) [80], [81] based framework.”