Movie Description

By Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, C. Pal, Hugo Larochelle, Aaron C. Courville, Bernt Schiele. 2016

423 citations recorded by the source corpus.

Audio description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. We introduce the Large Scale Movie Description Challenge (LSMDC) which contains a parallel corpus of 128,118 sentences aligned to video clips from 200 movies (around 150 h of video in total). The goal of the challenge is to automatically generate descriptions for the movie clips. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in the challenges organized in the context of two workshops at ICCV 2015 and ECCV 2016.

Subfields: Civil and Structural Engineering, Computer Vision and Pattern Recognition, Language and Linguistics, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Civil and Structural Engineering

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Multimodal Learning

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “…corpus consists of 113k multiple choice questions (73k training, 20k validation, 20k test) and is derived from pairs of consecutive video captions from ActivityNet Cap-tions (Krishna et al., 2017; Heilbron et al., 2015) and the Large Scale Movie Description Challenge (LSMDC; Rohrbach et al., 2017).”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “The missing domain for HellaSwag models is movie descriptions (LSMDC), still, Hella-Swag models obtain 69% accuracy.”
Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “ional MIL considers only one positive (orange circle) while discarding the rest. speech. Numerous prior works [18,21,27,28,42,51,55, 60,62,88,79,80,83,84] have used image / video description datasets [48,63,65,87,90] to learn an embedding space where visual and textual data are close only if they ar”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “We provide experimental results for a variety of domains ranging from instructional videos in CrossTask [68], cooking videos in YouCook2 [67], generic YouTube videos in MSR-VTT [58] to movie video clips in LSMDC [44].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “We report results on 12 video reasoning tasks: TVQA [64], TVQA(+) [65], VLEP [66], MSRVTT-QA [117], MSRVTT-Multichoice [121], LSMDC-Multichoice, LSMDC ﬁll-in-the-blank QA [110, 92], ActivityNetQA [122, 45], TGIFQA [52], and DramaQA [23].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “…the web; and finally (iv) we achieve state-of-the-art performance by only using the video modality on MSR-VTT [72], MSVD [10], DiDeMo [3] and LSMDC [53] – outperforming works that use pre-extracted experts from multiple modalities, as well as those that are pretrained on the noisy HowTo100M, which…”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “r, and four mismatched descriptions. For text-video clip retrieval, following [49], we use 1,000 pairs text-video for evaluation. LSMDC: We evaluate ﬁll-in-the-blank video question answering on LSMDC [32]. This task is to predict a single answer given a video clip and a sentence with a blank in it. ”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Various datasets for video description/captioning have been introduced to empower different ways to describe the video content, covering a wide range of domains, such as cooking [16, 68, 44, 45], movie [54, 46, 47], social media [22], and human activities [14, 50, 63, 30].”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “l classiﬁer for determining which images were interesting. Our ﬁltering process evolved as we collected data for the task. The ﬁrst author of this paper ﬁrst manually annotated 2000 images from LSMDC [67] as being ‘interesting’ or ‘not interesting’ and trained a logistic regression model to predict ”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “We defer more details of the dataset and challenge rules to [1] and the homepage(1).”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, methodology): “…including image cap-tioning datasets, such as Flickr30k (Young et al., 2014), MSCOCO (Lin et al., 2014), Conceptual Captions (Sharma et al., 2018), as well as video captioning datasets including LSMDC (Rohrbach et al., 2017), ActivityNet (Krishna et al., 2017), and VATEX (Wang et al., 2019b).”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “In the video domain, the LSMDC-QA dataset [30] introduced the movie ﬁll-in-the-blank task by adding action verbs to the answer set, requiring spatio-temporal reasoning from videos at the sequence level (similar to action recognition).”
Localizing Moments in Video with Natural Language (2017, methodology): “Table 2 contrasts the kinds of words used in DiDeMo to two natural language object retrieval datasets [17, 22] and two video description datasets [33, 50].”
Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, methodology): “LSMDC16 [18] movie description dataset with 101,079 training, 10,053 test and 7,408 validation videos.”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “For a more thorough comparison of video-text datasets, see [31] Figure 2 is the histogram of responses (blanked-out words) for the training set, showing that most words occur 100-200 times, with a heavy tail of more frequent words going up to 12,541 for the most frequent word (her).”
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (2016, methodology): “In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in all the four tasks of LSMDC 2016 [18].”
MarioQA: Answering Questions by Watching Gameplay Videos (2016, methodology): “Video question answering (VideoQA) is a more challenging task and is recently introduced as a natural extension of ImageQA in [13, 18, 29].”