Bidirectional Multirate Reconstruction for Temporal Modeling in Videos

By Linchao Zhu, Zhongwen Xu, Yi Yang. 2016

78 citations recorded by the source corpus.

Despite the recent success of neural networks in image feature learning, a major problem in the video domain is the lack of sufficient labeled data for learning to model temporal information. In this paper, we propose an unsupervised temporal modeling method that learns from untrimmed videos. The speed of motion varies constantly, e.g., a man may run quickly or slowly. We therefore train a Multirate Visual Recurrent Model (MVRM) by encoding frames of a clip with different intervals. This learning process makes the learned model more capable of dealing with motion speed variance. Given a clip sampled from a video, we use its past and future neighboring clips as the temporal context, and reconstruct the two temporal transitions, i.e., present-past transition and present-future transition, reflecting the temporal information in different views. The proposed method exploits the two transitions simultaneously by incorporating a bidirectional reconstruction which consists of a backward reconstruction and a forward reconstruction. We apply the proposed method to two challenging video tasks, i.e., complex event detection and video captioning, in which it achieves state-of-the-art performance. Notably, our method generates the best single feature for event detection with a relative improvement of 10.4% on the MEDTest-13 dataset and achieves the best performance in video captioning across all evaluation metrics on the YouTube2Text dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, neutral): “, image search and video understanding [33].”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “The dominant approach for multi-modal modeling is to leverage Recurrent Neural Networks (RNNs) and their variants, e.g ., Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), to model sequence relations, e.g ., [29, 55].”
Searching for a Robust Neural Architecture in Four GPU Hours (2019, background): “Designing an efficient and effective neural architecture requires substantial human effort and takes a long time [7, 9, 10, 12, 14, 20, 37, 44].”
Position Focused Attention Network for Image-Text Matching (2019, methodology): “As for the words, the final feature is obtained by feeding the embedding vector into a bi-directional GRU [Zhu et al ., 2017], whose dimension of the hidden state is also set as h.”
Contrastive Adaptation Network for Unsupervised Domain Adaptation (2019, background): “Recent advancements in deep neural networks have successfully improved a variety of learning problems [40, 8, 26, 19, 20].”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, unknown): “mGRU [62] 33.”
Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning (2019, unknown): “8 mGRU [38] 53.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, background): “Some works [37,4,12,78,61] tried to improve the encoder by considering the temporal structures inside videos.”
Compound Memory Networks for Few-Shot Video Classification (2018, methodology): “Recurrent Neural Networks have also been utilized to model video sequences [38, 39].”