A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering

By Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C. Courville, C. Pal. 2016

113 citations recorded by the source corpus.

While deep convolutional neural networks frequently approach or exceed human-level performance in benchmark tasks involving static images, extending this success to moving images is not straightforward. Video understanding is of interest for many applications, including content recommendation, prediction, summarization, event/object detection, and understanding human visual perception. However, many domains lack sufficient data to explore and perfect video models. In order to address the need for a simple, quantitative benchmark for developing and understanding video, we present MovieFIB, a fill-in-the-blank question-answering dataset with over 300,000 examples, based on descriptive video annotations for the visually impaired. In addition to presenting statistics and a description of the dataset, we perform a detailed analysis of 5 different models predictions, and compare these with human performance. We investigate the relative importance of language, static (2D) visual features, and moving (3D) visual features, the effects of increasing dataset size, the number of frames sampled, and of vocabulary size. We illustrate that: this task is not solvable by a language model alone, our model combining 2D and 3D visual information indeed provides the best result, all models perform significantly worse than human-level. We provide human evaluation for responses given by different models and find that accuracy on the MovieFIB evaluation corresponds well with human judgment. We suggest avenues for improving video models, and hope that the MovieFIB challenge can be useful for measuring and encouraging progress in this very interesting field.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “We evaluate our model on Movie Fill-in-the-Blank [95, 81] task, which based on descriptive audio description for the visually impaired.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “LSMDC FiTB QA [76, 92] The Fill-in-the-blank (FiTB) task is, given a video clip and a sentence with a blank in it, to predict a single correct word for the blank.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Early VideoQA benchmarks [31, 51, 53, 54, 59, 64] rely on video descriptions [27, 52] (e.”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “Text-only BLSTM [23] 32.0 Text-only Human [23] 30.2 GoogleNet-2D + C3D [23] 35.7 Merging-LSTM [24] 34.2 SNUVL [50] 38.0 CT-SAN [51] 41.9 LR/RL LSTMs [25] 40.9 JSFusion [49] 45.5 ActBERT 48.6”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2016b) and video question answering (Maharaj et al., 2017; Lei et al., 2020a) to video captioning (Zhou et al.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “Recent work also includes movie or TV-clip based QA [75, 51, 46].”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
TutorialVQA: Question Answering Dataset for Tutorial Videos (2019, background): “Accordingly, the answer format needs to also be improved towards more ﬂexible ways than multiple choice (Tapaswi et al., 2016; Jang et al., 2017) or ﬁll-in-the-blank questions (Maharaj et al., 2017; Kim et al., 2017).”
Learning Fashion Compatibility with Bidirectional LSTMs (2017, background): “Recently, several fill-in-the-blank (FITB) datasets [14, 15, 31, 32] have been created and evaluated to bridge visual and semantic information.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “Know-IT-VQA [39] 12,087 24,282 Manual Big Bang Theory Multiple-choice Movie-FIB [42] 118,507 348,998 Automatic LSMDC 2016 Open-ended”
Video Fill In the Blank Using LR/RL LSTMs with Spatial-Temporal Attentions (2017, methodology): “We perform experiments on two datasets: the original LSMDC Dataset [27] to evaluate the single blank VFIB problem (i.”
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, background): “Prime examples include datasets for image or video captioning [10, 38, 51, 24, 56, 28], question answering (QA) [5, 13, 54, 58, 19, 36, 46, 26], as well as audio-visual dialog [11, 1].”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Existing Video QA datasets include MovieFIB [44], MovieQA [54], TGIF-QA [26], PororoQA [32], and TVQA [35, 36].”
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020, background): “Hence, it has become important to develop automated models that can accurately extract such precise multimodal information from videos (Tapaswi et al., 2016; Maharaj et al., 2017; Kim et al., 2017; Jang et al., 2017; Gao et al., 2017; Anne Hendricks et al., 2017; Lei et al., 2018, 2020).”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Some early work have attempted to fill this gap by introducing datasets that leveragemovie data (Maharaj et al. 2017; Rohrbach et al. 2017; Tapaswi et al. 2016), focusing on storyline comprehension either from short video clips (Maharaj et al.”
Are we Asking the Right Questions in MovieQA? (2019, unknown): “This includes movie descriptions [18], video understanding through fill in the blank [13], video retrieval [20], character co-referencing [17] and image captioning [23].”