MovieQA: Understanding Stories in Movies through Question-Answering

By Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, Sanja Fidler. 2015

830 citations recorded by the source corpus.

We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text. The dataset consists of 14,944 questions about 408 movies with high semantic diversity. The questions range from simpler "Who" did "What" to "Whom", to "Why" and "How" certain events occurred. Each question comes with a set of five possible answers, a correct one and four deceiving answers provided by human annotators. Our dataset is unique in that it contains multiple sources of information - video clips, plots, subtitles, scripts, and DVS. We analyze our data through various statistics and methods. We further extend existing QA techniques to show that question-answering with such open-ended semantics is hard. We make this data set public along with an evaluation benchmark to encourage inspiring work in this challenging domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The NarrativeQA Reading Comprehension Challenge (2017, methodology): “In computer vision, the MovieQA dataset (Tapaswi et al., 2016) fulﬁlls a similar role as Nar-rativeQA.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “CLEVR, our diagnostic dataset, is most closely related to benchmarks for visual question answering [4, 10, 21, 26, 32, 37, 46, 49], as it involves answering natural-language questions about images.”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Visual Question Answering Visual Question Answering [1, 36] requires a model to answer visual related questions understanding both visual content and language.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “MovieQA [42] and TVQA [25] may also invoke causal and temporally related questions.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [12, 36, 37, 43, 44, 46, 74, 80, 87], our video representation is exclusively based on visual information, as we focus on the visual understanding of videos.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “There are many existing video-and-language tasks to evaluate the model’s capacities in joint video-text representation learning, e.g., video question answering [38, 10, 19, 56], video captioning [48, 54], text-video retrieval [49, 43, 26], video grounding [52].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “For example, MovieFIB (Maharaj et al., 2017) focuses on visual concepts, MovieQA (Tapaswi et al., 2016) is based on text summaries, and TGIF-QA(Jang et al., 2017) uses predefined templates for question generation on short GIFs.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, methodology): “Compared to these datasets, we have a comparable number of questions to DAQUAR [32] as well as MovieQA [41], and many more questions than knowledge-based datasets KB-VQA [43] and FVQA [44].”
Visual Dialog (2016, background): “tion [55], object detection [27] – to ‘high-level’ AI tasks such as learning to play Atari video games [35] and Go [47], answering reading comprehension questions by understanding short stories [15, 57], and even answering questions about images [4, 32, 41] and videos [49, 50]!”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “However, although many tasks combining visual and linguistic reasoning have been motivated by their potential robotic applications [4, 14, 21, 30, 42], none of these tasks allow an agent to move or control the camera.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “While past work has asked humans to write several counterfactual choices for each correct answer [75, 46], this process is expensive.”
Machine Comprehension Using Match-LSTM and Answer Pointer (2016, methodology): “We thank Pranav Rajpurkar for testing our model on the hidden test dataset and Percy Liang for helping us with the Dockerfile for Codalab.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, neutral): “Recent work on VQA includes datasets [32, 2, 23, 81, 41, 57, 47, 19, 66] and models [33, 60, 29, 31, 1, 67, 17, 37].”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, background): “Fortunately, large-scale video QA datasets have been recently emerged from the community using crowdsourcing on various sources of data (e.g. movies for MovieQA [10] and animated GIFs for TGIF-QA [11]).”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “MovieQA (Tapaswi et al., 2016) is most similar to our dataset, with both multiple choice questions and timestamp annotation.”
Movie Description (2016, methodology): “Tapaswi et al. (2016) used our AD transcripts for building their MovieQA dataset, which asks natural language questions about movies, requiring an understanding of visual and textual information, such as Dialogue and AD, to answer the question.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “Like the CLEVR dataset, benchmark datasets for VQA typically comprise a set of questions on images with associated answers [3, 31, 39, 24, 50]; both questions and answers are generally posed in natural language.”