MarioQA: Answering Questions by Watching Gameplay Videos

By Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, Bohyung Han. 2016

114 citations recorded by the source corpus.

We present a framework to analyze various aspects of models for video question answering (VideoQA) using customizable synthetic datasets, which are constructed automatically from gameplay videos. Our work is motivated by the fact that existing models are often tested only on datasets that require excessively high-level reasoning or mostly contain instances accessible through single frame inferences. Hence, it is difficult to measure capacity and flexibility of trained models, and existing techniques often rely on adhoc implementations of deep neural networks without clear insight into datasets and models. We are particularly interested in understanding temporal relationships between video events to solve VideoQA problems; this is because reasoning temporal dependency is one of the most distinct components in videos from images. To address this objective, we automatically generate a customized synthetic VideoQA dataset using Super Mario Bros. gameplay videos so that it contains events with different levels of reasoning complexity. Using the dataset, we show that properly constructed datasets with events in various complexity levels are critical to learn effective models and improve overall performance.

Subfields: Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Specifically, MarioQA [28] and COG [40] generate synthetic questions with simulated environments.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “These approaches have been extended to the video domain as well [20, 34, 42].”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “Visual Question Answering: Several image-based VQA datasets have recently been constructed, e.g., DAQUAR (Malinowski and Fritz, 2014), VQA (Antol et al., 2015), COCO-Q (Ren et al., 2015a), FM-IQA (Gao et al., 2015), Visual Madlibs (Yu et al., 2015), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “, 2017), MarioQA (Mun et al., 2017), and TVQA (Lei et al.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, background): “A temporal attention mechanism is leveraged to selectively attend to one or more periods of a video in [16, 24, 35].”
Visual Reference Resolution using Attention Memory for Visual Dialog (2017, methodology): “Beyond VQA, attention models are used to find visual groundings of linguistic expressions in a variety of other multi-modal tasks, such as image captioning [3, 4], VQA in videos [22], and visual attributes prediction [28].”
BERT Representations for Video Question Answering (2020, background): “Due to the temporal nature of videos, video-QA presents some unique challenges with respect to standard VQA: 1) it requires to understand the temporal coherence in a set of frames [49, 28], and 2) it often needs to consider plot-related questions based on the associated subtitles [21, 18, 8].”
FigureQA: An Annotated Figure Dataset for Visual Reasoning (2017, result): “This is in line with the recent trend in visual-textual datasets, such as those for intuitive physics and reasoning [17, 18].”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Similar to our dataset, the MarioQA dataset (Mun et al. 2017) recently introduced gameplay-based QA tasks requiring spatio-temporal reasoning at the frame level.”
LifeQA: A Real-life Dataset for Video Question Answering (2020, methodology): “LSMDC, TGIF-QA, PororoQA, and MarioQA consist of short video clips (just a few seconds), which is difﬁcult to understand what is going on in a scene beyond several actions that can be identiﬁed.”
KnowIT VQA: Answering Knowledge-Based Questions about Videos (2019, background): “Video Question Answering VideoQA addresses specific challenges with respect to the interpretation of temporal information in videos, including action recognition (Maharaj et al. 2017; Jang et al. 2017; Zellers et al. 2019; Mun et al. 2017), story understanding (Tapaswi et al.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[36] Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han.”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “Some video-only question-answering benchmarks are synthetically generated [56, 43], which affords the granular control necessary to measure model abilities like causality [56], or counting [43].”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, background): “The main motivation behind creating MarioQA dataset [37] was to understand temporal relationships between video scenes to solve VQA problems.”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, background): “VideoQA is a relatively new task, which aims to provide accurate answer for a pair of a video clip and a natural language question [35, 41, 43].”