KnowIT VQA: Answering Knowledge-Based Questions about Videos

By Noa García, Mayu Otani, Chenhui Chu, Yuta Nakashima. 2019

97 citations recorded by the source corpus.

We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show. Our main findings are: (i) the incorporation of knowledge produces outstanding improvements for VQA in video, and (ii) the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”
Boosting Visual Question Answering with Context-aware Knowledge Aggregation (2020, background): “Several previous works [7, 16, 28] make use of unstructured text corpus (e.g., Wikipedia text, natural language sentences) as the source of external knowledge.”
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge (2022, background): “Other datasets have considered knowledgebased question answering for a sitcom [10] and by using web queries [6].”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “[27] Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima.”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, background): “Know-IT-VQA [39] 12,087 24,282 Manual Big Bang Theory Multiple-choice Movie-FIB [42] 118,507 348,998 Automatic LSMDC 2016 Open-ended”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[16] Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima.”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “Several works [23, 6, 5, 27, 24] have incorporated visual embeddings or captions in predicting the final answers.”
BERT Representations for Video Question Answering (2020, background): “…by a LSTM, whereas the same authors improve the results in [22] with a network that grounds evidence in both the spatial and temporal domains; in [8] a video understanding task by fusing external knowledge and video-QA is introduced; in [15] a video question answering framework that requires to…”
CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation (2025, unknown): “…and chart data, ManyModalQA considers text, images, and tables, WavCaps [42], and MusicCaps [1] emphasize audio-language understanding, KnowIT VQA [17] and SOK-Bench [64] focus on video reasoning, and InfoSeek [7] and Encyclopedic VQA [43] address knowledge-intensive, information-seeking tasks.”