OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
By Kenneth Marino, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi. 2019
1,612 citations recorded by the source corpus.
Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Sergey Levine
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Question Answering & Reading Comprehension
Citations in the corpus
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
- Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “We use345 the standard VQA 2.0 (Goyal et al., 2017), and346 also OK-VQA (Marino et al., 2019) which requires347 knowledge to answer questions correctly.348 NLU and NLG.”
- KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, background): “KAT sets a new state-of-the-art on the challenging OK-VQA (Marino et al., 2019) benchmark, and sig-nificantly outperforms existing approaches.”
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “…of structured, task-based datasets [42,56], we also opt to include the training sets of various downstream tasks – adding the train sets of VQA [2], GQA [23], OKVQA [31], VizWiz [17], Hateful Memes [25], and CoCo Captions [9], resulting in a final dataset of around 25 million image-text pairs.”
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “The problem of knowledge-based visual question answering (VQA) (Marino et al. 2019) extends the standard VQA task (Antol et al. 2015) by asking questions that require outside knowledge beyond the image content to answer.”
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “∗ Training images of GQA were observed during training. four additional datasets that are used in InstructBLIP: open-knowledge VQA (OKVQA [40], A-OKVQA [44]) and OCR (OCRVQA [41], TextCaps [46]).”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “We assess ViperGPT on the OK-VQA dataset [18], designed to evaluate models’ abilities to answer questions requiring extrinsic knowledge.”
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “PaLM-E-562B achieves state-of-the-art performance on the OK-VQA (Marino et al., 2019) benchmark, without relying on task-specific finetuning.”
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “On the OK-VQA (Marino et al., 2019) dataset, BLIP-2 comes secondary to Flamingo80B.”
- Visual Programming: Compositional visual reasoning without training (2022, methodology): “Instead of assembling an expensive dataset and training a multi-image model, we demonstrate the ability of VISPROG to use a single-image VQA system to solve a task involving multiple images without training on multi-image examples.”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Manning, 2019), and OK-VQA (Marino et al., 2019).”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “Knowledge-intensive tasks [40, 49] — those where a human can only be expected to perform the task with access to a knowledge source such as Wikipedia — are challenging for even cutting edge NLP and vision-and-language models, as it is infeasible to train large models to memorize everything.”
- A Generalist Agent (2022, background): “In particular OKVQA (Marino et al., 2019) and VQAv2 (Antol et al., 2015) with 9K and 443K triplets of images, questions, and answers.”
- Language Models are General-Purpose Interfaces (2022, methodology): “Visual Question Answering Following (Tsimpoukelli et al., 2021), we evaluate the zero-shot performance on VQAv2 (Goyal et al., 2017) validation set and OK-VQA (Marino et al., 2019) Table 8: Zero-shot image captioning results on NoCaps validation and Flickr30k test.”
- Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “Only on OK-VQA, Flamingo performs better than P N P-VQA.”
- KB-VLP: Knowledge Based Vision and Language Pretraining (2021, methodology): “Outside Knowledge Visual Question Answering (OK-VQA) (Marino et al., 2019) is a new dataset that asks models to draw upon outside knowledge to answer questions.”
- Generative Multimodal Models are In-Context Learners (2023, methodology): “Our Emu2-Chat is evaluated on academic-task-oriented benchmarks including image question-answering datasets (VQAv2 [28], OKVQA [53], GQA [32], VizWiz [29], TextVQA [68]) and video question-answering datasets (MSVD [82] and MSRVTT [82]).”
- ConceptBert: Concept-Aware Representation for Visual Question Answering (2020, methodology): “Recent research has started integrating knowledge-based methods into VQA models (Wang et al., 2017, 2016; Narasimhan et al., 2018; Narasimhan and Schwing, 2018; Zhu et al., 2015; Marino et al., 2019).”