Learnable pooling with Context Gating for video classification

By Antoine Miech, Ivan Laptev, Josef Sivic. 2017

344 citations recorded by the source corpus.

Common video representations often deploy an average or maximum pooling of pre-extracted frame features over time. Such an approach provides a simple means to encode feature distributions, but is likely to be suboptimal. As an alternative, we here explore combinations of learnable pooling techniques such as Soft Bag-of-words, Fisher Vectors , NetVLAD, GRU and LSTM to aggregate video features over time. We also introduce a learnable non-linear network unit, named Context Gating, aiming at modeling in-terdependencies between features. We evaluate the method on the multi-modal Youtube-8M Large-Scale Video Understanding dataset using pre-extracted visual and audio features. We demonstrate improvements provided by the Context Gating as well as by the combination of learnable pooling methods. We finally show how this leads to the best performance, out of more than 600 teams, in the Kaggle Youtube-8M Large-Scale Video Understanding challenge.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “the right-hand side in Equations (2) and (3) is a linear fullyconnected layer and the second term corresponds to a context gating function [31] with an output ranging between 0 and 1, which role is to modulate the output of the linear layer.”
Squeeze-and-Excitation Networks (2017, background): “The benefits of such a mechanism have been shown across a range of tasks, from localisation and understanding in images [3, 19] to sequence-based models [2, 28].”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “We start by considering the context feature gating mechanism first used for video classification in [23].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “The second layer, given by (2), performs context gating [34], where individual dimensions of Z1 are reweighted using learnt gating weights σ(W2Z1+b2) with values between 0 and 1, where W2 and b2 are learnt parameters.”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [10, 27, 28, 45] guided by human pose or other primitives [4, 5, 10, 11], recurrent models [34] and self-attention [44].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “We apply non-linear feature gating [30] enabling our model to re-calibrate the feature activations before the ﬁnal output embeddings.”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, background): “Recently, tentative works have been proposed to apply attention into deep neural networks [23, 26, 27], ranging from localization and understanding in images [28, 29] to sequence-based networks [30, 31].”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “[21] found that us-ing NetVLAD [3] results in an increase in accuracy over GRUs or LSTMs for aggregation of both visual and text features.”
Label Denoising with Large Ensembles of Heterogeneous Neural Networks (2018, methodology): “We used the VLADBoW model proposed in [7].”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, methodology): “Action recognition: Recent work on action recognition using deep learning involves learning compact (coarse) representations over time and use pooling or other aggregation methods to combine information from each video frame, or even across different modalities [10, 13, 30, 41, 43].”
Videos as Space-Time Region Graphs (2018, background): “To tackle this limitation, recent work has also focused on modeling long term temporal information with Recurrent Neural Networks [3,4,5,6] and 3D Convolutions [7,8,9,10].”
Efficient Video Classification Using Fewer Frames (2019, methodology): “In the training of NetVLAD model, we have used the standard hyperparameter settings as mentioned in [23].”
Composing Text and Image for Image Retrieval - an Empirical Odyssey (2018, methodology): “Inspired by [47, 14, 30], we propose to combine image and text features using the following approach which we call Text Image Residual Gating (or TIRG for short).”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Some propose learned strategies [25, 20], while others advocate pooling [40].”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, methodology): “classification[35, 17], and action recognition[28].”
Squeeze-and-Excitation Networks (2017, background): “Attention mechanisms have demonstrated their utility across many tasks including sequence learning [52], [53], localisation and understanding in images [9], [54], image captioning [55], [56] and lip reading [57].”
Anticipative Video Transformer (2021, background): “The typical approach to solving long-term predictive reasoning tasks involves extracting frame or clip level features using standard architectures [12, 83, 88], followed by aggregation using clustering [32, 60], recurrence [23, 24, 41], or attention [28, 57, 74, 92] based models.”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, background): “Finally, we do note that many other video models have been proposed in literature involving 2.5D convolutions (Tran et al., 2018; Xie et al., 2017), VLAD-style aggregation (Girdhar et al., 2017; Miech et al., 2017) and other multi-modal architectures (Wang et al., 2016a; Bian et al., 2017).”