Delving Deeper into Convolutional Networks for Learning Video Representations

By Nicolas Ballas, L. Yao, C. Pal, Aaron C. Courville. International Conference on Learning Representations, 2015

802 citations recorded by the source corpus.

We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset. While high-level percepts contain highly discriminative information, they tend to have a low-spatial resolution. Low-level percepts, on the other hand, preserve a higher spatial resolution from which we can model finer motion patterns. Using low-level percepts can leads to high-dimensionality video representations. To mitigate this effect and control the model number of parameters, we introduce a variant of the GRU model that leverages the convolution operations to enforce sparse connectivity of the model units and share parameters across the input spatial locations. We empirically validate our approach on both Human Action Recognition and Video Captioning tasks. In particular, we achieve results equivalent to state-of-art on the YouTube2Text dataset using a simpler text-decoder model and without extra 3D CNN features.

Subfields: Cancer Research, Computer Vision and Pattern Recognition

View Paper: https://arxiv.org/abs/1511.06432

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

Spatiotemporal Residual Networks for Video Action Recognition (2016, methodology): “Method UCF101 HMDB51 Two-Stream ConvNet [20] 88.0% 59.4% Two-Stream+LSTM[18] 88.6% - Two-Stream (VGG16) [1, 31] 91.4% 58.5% Transformations[31] 92.4% 62.0% Two-Stream Fusion[5] 92.5% 65.4% ST-ResNet* 93.4% 66.4% Method UCF101 HMDB51 IDT [29] 86.4% 61.7% C3D + IDT [26] 90.4% - TDD + IDT [30] 91.5%…”
Improving Interpretability of Deep Neural Networks with Semantic Information (2017, background): “For example, some works have shown the effectiveness of DNNs on video analysis [2, 35] when stacking a hierarchical RNN on top of some CNN layers.”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, background): “Many attempts in this genre leverage CNNs trained on images to extract features from the individual frames and then perform temporal integration of such features into a fixed-size descriptor using pooling, high-dimensional feature encoding [41, 11], or recurrent neural networks [42, 7, 32, 2].”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, background): “However, recently models such as [42, 10, 43, 4] have shifted toward a more general encoder-decoder neural approach to tackle the captioning problem for open domain videos.”
Semantic Compositional Networks for Visual Captioning (2016, background): “Representative works include [7, 9, 10, 20, 23, 24, 29, 48] for image captioning and [10, 46, 47, 56, 3, 35, 11] for video captioning.”
VideoLSTM convolves, attends and flows for action recognition (2016, methodology): “Following the previous works [31,32,33], we use the ﬁrst split and report mean average precision (mAP) over all 24 classes.”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “(10) ConvGRU is proposed as a convolutional counterpart to previous fully connected GRU [9], and introduces convolution operation into input-to-state and state-to-state transitions.”
Semantic Video Segmentation by Gated Recurrent Flow Propagation (2016, methodology): “ral features easier by using separable ﬁlters. Speciﬁcally, they ﬁrst apply spatial ﬁlters and then temporal ﬁlters in order to avoid learning different temporal combinations for every spatial ﬁlter. [1] learn spatiotemporal ﬁlters by stacking GRUs at different locations in a deep neural network and”
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model (2017, background): “TrajGRU allows the state to be aggregated along some learned trajectories and thus is more flexible than the Convolutional GRU (ConvGRU) [2] whose connection structure is fixed.”
Neural Networks and Deep Learning (2018, positive): “A recent method [21] combines recurrent and convolutional neural networks in a homogeneous way.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “(8)) are all implemented using ConvGRU[2], with 1×1 convolutional kernels.”
Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving (2020, methodology): “Specially, we consider the following four comparison methods: 1) ConvLSTM as used in [27], 2) ConvGRU as used in [1], 3) T-encoder which does not consider the channel-wise correlations by taking the whole representation of each frame as a node for encoding (e.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Besides using LSTM, some works [87], [88], [89], [90] have used Gated Recurrent Units (GRUs) [91] for HAR, which can also alleviate the vanishing gradient problem of”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “The ﬁgure also reports the captions for GRU-MP-(CI) and GRU-EVE hft -(CI) to show the difference resulting from hi-Table 29.2 LSTM-YT [50] 29.07 TA [57] 29.0 p-RNN [58] 31.1 HRNE [31] 33.1 BGRCN [5] 31.70 MAA [16] 31.80 RMA [22] 31.90 LSTM-E [32] 29.5 M 3 -inv3 [53] 32.18 GRU-EVE hft -(IRV2) 33.7 1.”
RPAN: An End-to-End Recurrent Pose-Attention Network for Action Recognition in Videos (2017, background): “However, the inputs to these LSTMs are high-level features obtained from the fully-connected (FC) layer of CNNs, which are limited to represent fine action details in videos [1].”
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition (2016, background): “While earlier works resorted to unsupervised learning of 3D spatio-temporal features [45], supervised end-to-end learning has recently gained popularity [11,12,13,14,15,16,17,18,19,20,21,22].”
Video Object Detection with an Aligned Spatial-Temporal Memory (2017, methodology): “Apart from video object detection, other recent work use convolutional and/or recurrent networks for video classification [26,42,4].”
Transformation-based Adversarial Video Prediction on Large-Scale Data (2020, methodology): “[5] introduce convolutional variants of the Long Short-Term Memory (LSTM) [27] and Gated Recurrent Unit (GRU) modules [12].”