A dataset for Movie Description

By Anna Rohrbach, Marcus Rohrbach, Niket Tandon, Bernt Schiele. 2015

569 citations recorded by the source corpus.

Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the MPII Movie Description dataset (MPII-MD) contains a parallel corpus of over 68K sentences and video snippets from 94 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are far more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Evaluation & Benchmarks

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “[30] recently released the Movie Description dataset which contains clips from movies, each time-stamped with a sentence from DVS (Descriptive Video Service).”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “This is the largest public benchmark [30] related to our work that considers movie data (but without person localization as we do in our work).”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, neutral): “Charades [48] 10k 16k 10,000 82h Home 2016 MSR-VTT [58] 10k 200k 7,180 40h Youtube 2016 YouCook2 [67] 14k 14k 2,000 176h Youtube 2018 EPIC-KITCHENS [7] 40k 40k 432 55h Home 2018 DiDeMo [15] 27k 41k 10,464 87h Flickr 2017 M-VAD [52] 49k 56k 92 84h Movies 2015 MPII-MD [43] 69k 68k 94 41h Movies 2015 A”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “LSMDC [3] is built upon 118K video clips from 202 movies.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “LSMDC [45] consists of 118,081 video clips sourced from 202 movies.”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “A few datasets have been introduced for this task and cover a variety of domains, such as cooking [16, 72], movie [47],”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “In datasets that are commonly used for retrieval [7,28,29,97], captions are considered relevant if they were collected for the same video, and irrelevant otherwise.”
Dense-Captioning Events in Videos (2017, background): “15 [15] sports 21k 4s - - - - - HMDB 51 [25] movie 7k 3s - - - - - Hollywood 2 [28] movie 4k 20s - - - - - MPII cooking [40] cooking 44 600s - - - - - ActivityNet [4] human 20k 180s - - - - - MPII MD [39] movie 68k 4s 68,375 X - - - M-VAD [47] movie 49k 6s 55,904 X - - - MSR-VTT [55] open 10k 20s 20”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “, 2017), LSMDC (Rohrbach et al., 2015), TRECVID (Over et al.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “Our dataset is unique in that it contains multiple sources of information – video clips, plots, subtitles, scripts, and DVS [32].”
Movie Description (2016, methodology): “…label semantics. b) Training one-vs-all SVMs for speciﬁc label groups consistently improves the performance as it avoids “wrong” negatives. c) Focusing on namely TACoS Multi-Level (Rohrbach et al. 2014), and showed the comparable performance to manually annotated SRs, see (Rohrbach et al. 2015c).”
Sequence to Sequence -- Video to Text (2015, methodology): “We report results on three video description corpora, namely the Microsoft Video Description corpus (MSVD) [3], the MPII Movie Description Corpus (MPII-MD) [28], and the Montreal Video Annotation Dataset (M-VAD) [37].”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, background): “Recent papers have also used movies to focus on the video description problem leading to several datasets such as MSVD[20], M-VAD [21], and MPII-MD [11].”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “A wide range of tasks based on real-life videos have been designed to test such ability, including text-to-video retrieval [75, 28, 54], video captioning [54, 75, 82], video question answering [74, 23, 33, 34], and video moment retrieval [2, 18, 35].”
Localizing Moments in Video with Natural Language (2017, background): “Other video description datasets include descriptions of short clips sourced from full length movies [32, 45].”
Describing Videos by Exploiting Temporal Structure (2015, methodology): “Other recent work [25] has explored the use of DVS annotated video for video description research and has under-scored the observation that DVS descriptions are typically much more relevant and accurate descriptions of the visual content of a video compared to movie scripts.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “We compare TGIF to four existing image and video description datasets: MS-COCO [20], M-VAD [34], MPII-MD [31], and LSMDC [34, 31].”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “Such performance even outperforms S2VT which combines M-VAD and MPII-MD [24] for training.”