Sequence to Sequence -- Video to Text

By Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond J. Mooney, Trevor Darrell, Kate Saenko. 2015

1,495 citations recorded by the source corpus.

Real-world videos often have complex dynamics, methods for generating open-domain video descriptions should be sensitive to temporal structure and allow both input (sequence of frames) and output (sequence of words) of variable length. To approach this problem we propose a novel end-to-end sequence-to-sequence model to generate captions for videos. For this we exploit recurrent neural networks, specifically LSTMs, which have demonstrated state-of-the-art performance in image caption generation. Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip. Our model naturally is able to learn the temporal structure of the sequence of frames as well as the sequence model of the generated sentences, i.e. a language model. We evaluate several variants of our model that exploit different visual features on a standard set of YouTube videos and two movie description datasets (M-VAD and MPII-MD).

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, background): “…based models that map an input sequence into another output sequence, called sequence-to-sequence models, have been successful in many problems such as machine translation (Bahdanau et al., 2014), speech recognition (Bahdanau et al., 2015) and video captioning (Venugopalan et al., 2015).”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “(a) Naive [7], where the hidden state of the answer decoder is initialized with the output of the video-question (VQ) encoder.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Another popular task is video captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and…”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “To effectively utilize the two modalities, text and video, we design a multimodal sequence-to-sequence model [55, 59] with the attention mechanism [5, 38] for VMT.”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “Consequently, [70] propose an LSTM to encode video frames into a fixed length vector before sentence generation with an LSTM.”
Dense-Captioning Events in Videos (2017, methodology): “Specifically, we compare our work with LSTM-YT [37], S2VT [38] and H-RNN [51].”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, background): “Recently, various video-language tasks have drawn a lot of interests in computer vision research [1,2,3], including video captioning [4,5,6,7,8,9], video question answering (QA) [10,11], and video retrieval for a natural language query [8,12,13].”
PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “When training the S2VT [43] model on the MSVD [3] dataset, PipeDream speeds up training by 3x compared to data-parallel BSP.”
Movie Description (2016, methodology): “The other submissions are S2VT (Venugopalan et al. 2015b), Temporal Attention (Yao et al. 2015) and Frame-Video-Concept Fusion (Shetty and Laaksonen 2015).”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “Although our model is based on a sequence-to-sequence model [35] to achieve simplicity, it can be improved in different ways, such as adopting the concept of 3D convolution [34].”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, background): “[42], which is a combination of a CNN, and a LSTM.”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “For example, LSTM-YT [32], S2YT [31], TempoAttn [36], H-RNN [41] and DEM [20].”
Localizing Moments in Video with Natural Language (2017, background): “Contemporary models integrate deep video representations with recurrent language models [25, 31, 47, 48, 56].”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “…framework does not constrain the input or target sequences, and has been successfully applied to tasks such as machine translation [21, 22], video caption generation [23], abstract meaning representation parsing and generation [24], and acoustic word embeddings acquisition [11].”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “This conforms to recent ﬁndings that end-to-end sequence learning using deep neural nets outperforms traditional hand-crafted pipelines [43, 38].”
Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, background): “In the video-language domain, the few works that exist focus largely on video description generation [25,31].”
Improving Interpretability of Deep Neural Networks with Semantic Information (2017, methodology): “…Model BLEU METEOR LSTM-B (GoogLeNet) 0.416 0.295 LSTM-I (GoogLeNet) 0.446 0.297 LSTM-YT (AlexNet) [32] 0.333 0.291 S2VT (RGB + Optical Flow) [31] - 0.298 SA (GoogLeNet) [34] 0.403 0.290 LSTM-E (VGG) [26] 0.402 0.295 h-RNN (VGG) [35] 0.443 0.311 SA (GoogLeNet + C3D) [34] 0.419 0.296 LSTM-E…”