ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification

By Rohit Girdhar, Deva Ramanan, Abhinav Kumar Gupta, Josef Sivic, Bryan C. Russell. 2017

470 citations recorded by the source corpus.

In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video. We do so by integrating state-of-the-art two-stream networks [42] with learnable spatio-temporal feature aggregation [6]. The resulting architecture is end-to-end trainable for whole-video classification. We investigate different strategies for pooling across space and time and combining signals from the different streams. We find that: (i) it is important to pool jointly across space and time, but (ii) appearance and motion streams are best aggregated into their own separate representations. Finally, we show that our representation outperforms the two-stream base architecture by a large margin (13% relative) as well as outperforms other baselines with comparable base architectures on HMDB51, UCF101, and Charades video classification benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “Pooling is a well-known technique to combine various temporal features; concatenation, averaging, maximum, minimum, ROI, feature aggregation techniques and time-domain convolution are some of the possible pooling techniques [11,21].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “For video, previous works also employ self-attention blocks together with CNN layers, for action recognition [17] and video classification [10].”
Video Representation Learning Using Discriminative Pooling (2018, neutral): “…(mAP) Method Classiﬁcation Detection Two-stream VGG (Average Pooling) [40] 14.3% 10.9% Two-stream VGG (Max Pooling) [40] 15.3% 9.2% ActionVLAD + IDT[16] 21.0% - Asynchronous Temporal Fields [38] 22.4% 12.8% SVMP(VGG) 25.1% 13.9% SVMP(VGG+IDT) 26.7% 14.2% NTU-RGBD Method Cross-Subject Cross-View…”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “Finally, we note that the longrange modeling approach of TimeSformer is orthogonal to long-term schemes designed to operate on top of video backbones (Wu et al., 2019; Girdhar et al., 2017), and thus, further gains may come from a combination of them.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Most prior works [12, 31, 33, 37, 38] address the problem of appearance and motion fusion for video representation.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “An alternative way to model the temporal relationships is to use 2D CNN + posthoc fusion [13, 9, 58, 7].”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [11, 29, 30, 48], guided by human pose or other primitives [4, 5, 11, 12], regiongraph representations [18, 47], recurrent models [36] and self-attention [46].”
Learnable pooling with Context Gating for video classification (2017, methodology): “This type of end-to-end trainable orderless aggregation has been recently applied to video frames in [26].”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, background): “Action recognition: Recent work on action recognition using deep learning involves learning compact (coarse) representations over time and use pooling or other aggregation methods to combine information from each video frame, or even across different modalities [10, 13, 30, 41, 43].”
Dual Contrastive Prediction for Incomplete Multi-View Representation Learning (2022, background): “(2) VLAD (Action Vector of Local Aggregated Descriptor) [71] learns the representation by aggregating local features and the video spatio-temporal content.”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work includes capturing long-term temporal structure via recurrent networks [74, 11] or ranking functions [17], pooling across space and/or time [63, 23], modeling hierarchical or spatiotemporal information in videos [44, 58], building long-term temporal relations [67, 77], or boosting accura”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, background): “Recently, significant progress has been achieved in terms of the recognition performance in this area following the deep convolutional network paradigm [30, 5, 4, 6, 26, 1, 8, 21].”
Video Multitask Transformer Network (2019, methodology): “Another way, aggregating features using NetVLAD [1], which used for place recognition, performed well in action recognition and video classiﬁcation [10, 29].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, methodology): “VLAD [15] and NetVLAD [1] have achieved great impacts in aggregating discriminative features for video classification [10, 36], video retrieval [24], person re-identification [40].”
TEA: Temporal Excitation and Aggregation for Action Recognition (2020, methodology): “either 1) adopt 2D CNN backbones to extract frame-wise features and then utilize a simple temporal max/average pooling to obtain the whole video representation [42, 11].”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Especially for long snippets, it is often assumed that a learned representation is necessary [11, 20], though their eﬀectiveness over simple pooling is still controversial [40].”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, methodology): “0 – ActionVLAD [17] RGB 16.”