Describing Videos by Exploiting Temporal Structure

By L. Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, C. Pal, Hugo Larochelle, Aaron C. Courville. 2015

1,108 citations recorded by the source corpus.

Recent progress in using recurrent neural networks (RNNs) for image description has motivated the exploration of their application for video description. However, while images are static, working with videos requires modeling their dynamic temporal structure and then properly integrating that information into a natural language description model. In this context, we propose an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions. First, our approach incorporates a spatial temporal 3-D convolutional neural network (3-D CNN) representation of the short temporal dynamics. The 3-D CNN representation is trained on video action recognition tasks, so as to produce a representation that is tuned to human motion and behavior. Second we propose a temporal attention mechanism that allows to go beyond local temporal modeling and learns to automatically select the most relevant temporal segments given the text-generating RNN. Our approach exceeds the current state-of-art for both BLEU and METEOR metrics on the Youtube2Text dataset. We also present results on a new, larger and more challenging dataset of paired video and natural language descriptions.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “(c) An attention variant of the naive implementation, in which the attention can either be added to the question (AttQns) or the video (AttVid) [10] side.”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “[71] propose a similar model, adding a temporal attention mechanism which weights video frames differently when generating each word in a sentence.”
Theano: A Python framework for fast computation of mathematical expressions (2016, methodology): “[29] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville, “Describing videos by exploiting temporal structure,” in Computer Vision (ICCV), 2015 IEEE International Conference on (IEEE, 2015).”
Dense-Captioning Events in Videos (2017, background): “To avoid this issue, others have proposed either a recurrent encoder [6, 37, 42] or an attention mechanism [48].”
Movie Description (2016, methodology): “With respect to the sentence length, Visual-Labels and S2VT demonstrate similar properties to the reference descriptions, while the approaches Frame-Video-Concept Fusion and Temporal Attention generate much shorter sentences (5.16 and 3.63 words on average vs. 8.74 of the references).”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “e vector of the LSTM) to each candidate feature vector, embeds the resulting vectors into scalar values, and then applies the softmax function to generate the attention weight for each candidate. [19][20][21][23] use the \concatenation" alignment function in their soft attention models and [24]”
Sequence to Sequence -- Video to Text (2015, methodology): “Thus, the closest comparison between the Soft Attention Model [41] and our S2VT is arguably ours with VGG (line 10) vs. their GoogleNet only model (line 6).”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, methodology): “Unlike previous work that employs attention models in the 2D spatial and/or temporal dimension [48, 56], we explore its effect in the scale dimension.”
Grounding of Textual Phrases in Images by Reconstruction (2015, methodology): “[36] relies on a similar mechanism to perform temporal attention for selecting frames in video description task.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, background): “This conforms to recent ﬁndings that end-to-end sequence learning using deep neural nets outperforms traditional hand-crafted pipelines [43, 38].”
Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, background): “In the video-language domain, the few works that exist focus largely on video description generation [25,31].”
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, methodology): “The image captioning methods developed in [9, 14, 21, 28, 32], for example, learn a mapping from images to text using CNNs trained on object recognition, and word embeddings trained on large scale text corpora.”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “As it has been shown that characterizing entities in addition of action is important for the caption-generation task Yao et al. (2015a), we also use as encoder a CNN Szegedy et al. (2014), pretrained on ImageNet, that focuses on detecting static visual object categories.”