Translating Videos to Natural Language Using Deep Recurrent Neural Networks

By Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, Kate Saenko. 2014

989 citations recorded by the source corpus.

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “These approaches have also been extended to generate descriptions of short video clips [39].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “[69] propose creating a video feature by pooling high level CNN features across frames.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “On the video captioning task, the performance is evaluated on MSVD [9] with the widely-used splits from [102], MSRVTT [114], YouCook2 [131] (results in supplementary materials.”
Dense-Captioning Events in Videos (2017, methodology): “Specifically, we compare our work with LSTM-YT [37], S2VT [38] and H-RNN [51].”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “Another strand of research on multi-modal embeddings is based on deep learning [3, 24, 25, 31, 35, 44], utilizing such techniques as deep Boltzmann machines [44], autoencoders [35], LSTMs [8], and recurrent neural networks [31, 45].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In [10, 41], the authors caption video clips using LSTMs, [33] formulates description as a machine translation model, while older work uses templates [3, 8, 18].”
Movie Description (2016, methodology): “The other submissions are S2VT (Venugopalan et al. 2015b), Temporal Attention (Yao et al. 2015) and Frame-Video-Concept Fusion (Shetty and Laaksonen 2015).”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “However, the SMem-VQA Two-Hop model achieves an improvement of 2.27% on test-dev and 2.35% done in [32].”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “In computer vision, along with improvements in deep learning based visual recognition, there has been an explosion of recent interest in methods to automatically generate natural language descriptions for images [5, 9, 15, 32, 16, 20] or videos [31, 8].”
Sequence to Sequence -- Video to Text (2015, methodology): “In order to compare features from the VGG network with previous models, we include the performance of the mean-pooled model proposed in [39] using the output of the fc7 layer from the 16 layer VGG as a baseline (line 3, Table 2).”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “For example, LSTM-YT [32], S2YT [31], TempoAttn [36], H-RNN [41] and DEM [20].”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, neutral): “S2VT [65] 11.”
Localizing Moments in Video with Natural Language (2017, background): “[48] for video description and [46] for whole video retrieval).”
Describing Videos by Exploiting Temporal Structure (2015, methodology): “The architecture choice for the encoder φ depends on the type of input.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “[39] represent video by mean-pooling image features from frames, while Li et al .”