Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

By Atousa Torabi, C. Pal, Hugo Larochelle, Aaron C. Courville. 2015

216 citations recorded by the source corpus.

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, neutral): “Charades [48] 10k 16k 10,000 82h Home 2016 MSR-VTT [58] 10k 200k 7,180 40h Youtube 2016 YouCook2 [67] 14k 14k 2,000 176h Youtube 2018 EPIC-KITCHENS [7] 40k 40k 432 55h Home 2018 DiDeMo [15] 27k 41k 10,464 87h Flickr 2017 M-VAD [52] 49k 56k 92 84h Movies 2015 MPII-MD [43] 69k 68k 94 41h Movies 2015 A”
Dense-Captioning Events in Videos (2017, background): “and Charades [43] broadened the domain of activities captured by these datasets by including a large set of human activities. In an effort to map video semantics with language, MPII MD [39] and M-VAD [47] released short movie clips with descriptions. In an effort to capture longer events, MSRVTT [55”
Movie Description (2016, methodology): “In Section 3.2 we discuss how to fully automate AD audio segmentation and alignment for the Montreal Video Annotation Dataset (M-VAD), initially presented by Torabi et al. (2015).”
A dataset for Movie Description (2015, methodology): “This approach achieves competitive performance on TACoS Multi-Level corpus [55] without using the annotations and outperforms the retrieval approaches on our novel movie description dataset.”
Sequence to Sequence -- Video to Text (2015, methodology): “We report results on three video description corpora, namely the Microsoft Video Description corpus (MSVD) [3], the MPII Movie Description Corpus (MPII-MD) [28], and the Montreal Video Annotation Dataset (M-VAD) [37].”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, neutral): “Recent papers have also used movies to focus on the video description problem leading to several datasets such as MSVD[20], M-VAD [21], and MPII-MD [11].”
Localizing Moments in Video with Natural Language (2017, background): “Other video description datasets include descriptions of short clips sourced from full length movies [32, 45].”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “In essence, movie description and animated GIF description tasks both involve translating image sequence to natural language, so the LSMDC dataset may seem similar to the dataset proposed in this paper.”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “Two widely-used video captioning datasets, the Microsoft Research Video Description Corpus (MSVD) [5] and the Montreal Video Annotation Dataset (M-VAD) [34], are used in our experiments, which demonstrate the effectiveness of the proposed method.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “In addition, two large-scale movie description datasets, Montreal Video Annotation Dataset (M-VAD) [27] and MPII Movie Description Corpus (MPII-MD) [20], are included for evaluation on sentence generation.”
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks (2015, methodology): “2) Experimental Result: In [23], this approach to video description generation has been tested on two datasets; (1) Youtube2Text [54] and (2) Montreal DVS [55].”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “Detailed labeling has been addressed also in various video captioning datasets recently, where the goal is to predict an elaborate description, rather than a single label, for a video [31, 24, 37, 13].”
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2016, methodology): “The LSMDC2016 dataset [31, 39, 29] forms the basis of our proposed fill-in-the-blank dataset (MovieFIB) and evaluation.”
Video Captioning and Retrieval Models with Semantic Attention (2016, methodology): “and Montreal Video Annotation Dataset (M-VAD) [20].”
Video Captioning with Transferred Semantic Attributes (2016, unknown): “LSTM-TSA IV utilizing attributes learnt from images and videos signiﬁcantly (a) M-VAD dataset.”
Adaptive Feature Abstraction for Translating Video to Text (2016, methodology): “os for validation, and 670 videos for testing. The M-VAD is a large-scale movie description dataset, which is composed of 46587 movie snippets annotated with 56752 sentences. We follow the setting in Torabi et al. (2015), taking 36920 videos for training, 4950 videos for validation, and 4717 videos ”