A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
By Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, Roozbeh Mottaghi. 2022
950 citations recorded by the source corpus.
The Visual Question Answering (VQA) task aspires to provide a meaningful testbed for the development of AI models that can jointly reason over visual and natural language inputs. Despite a proliferation of VQA datasets, this goal is hindered by a set of common limitations. These include a reliance on relatively simplistic questions that are repetitive in both concepts and linguistic structure, little world knowledge needed outside of the paired image, and limited reasoning required to arrive at the correct answer. We introduce A-OKVQA, a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. In contrast to the existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image. We demonstrate the potential of this new dataset through a detailed analysis of its contents and baseline performance measurements over a variety of state-of-the-art vision-language models. Project page: http://a-okvqa.allenai.org/
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Yu Qiao
- Caiming Xiong
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yu Qiao
- Cordelia Schmid
In Multimodal Learning
In Question Answering & Reading Comprehension
- Peng Gao
- Chunyuan Li
- Haotian Liu
- Zhengyuan Yang
- Yong Jae Lee
- Aniruddha Kembhavi
- Hongsheng Li
- Victor Sanh
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “∗ Training images of GQA were observed during training. four additional datasets that are used in InstructBLIP: open-knowledge VQA (OKVQA [40], A-OKVQA [44]) and OCR (OCRVQA [41], TextCaps [46]).”
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “The POPE benchmark aggregates data from three distinct sources: MSCOCO [40], A-OKVQA [55], and GQA [27].”
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, neutral): “Sure,hereisapoemabouttheimageyouprovided:”
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “…pairs from 165k images by prompting 21], Infograph-icVQA train [22], AI2D train (transparent and opaque label boxes) [13], A-OKVQA train [28], Android-Control train [14], ScienceQA train [16], TabMWP train [17], ST-VQA train [6], TallyQA train [3], DVQA train [11], FigureQA train [12],…”
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “…script, we utilize the developed LLMs like GPT-4 [44] and Claude-2 [3] to generate movie summaries, plot-related QA pairs, and general reasoning QA pairs. tions from ShareGPT [1], 625K single-or multi-turn visual QA pairs from [19, 22, 25, 26, 33, 40–42, 47, 49], and 98K video QA pairs from [7].”
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, positive): “We uniformly sampled a subset of 300 questions from the dataset.”
- mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “∈ R d × d learnable projection matrices, and C l ∈ R ( L V + L T ) × d context features of l -th layer.”
- PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “Our system achieves state-of-the-art performance on OKVQA and A-OKVQA, and outperforms strong baselines in multimodal in-context learning on VQAv2 and TextVQA.”
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, background): “…with reading comprehension [13], visual reasoning [14, 15, 16], image question answering [17, 18], knowledge-grounded image question answering [19, 20, 21], image question answering with reading comprehension [22, 23], image question generation (adapted from the QA datasets), video question…”
- LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “A-OKVQA [48] emphasizes exter-nal knowledge beyond visible content.”
- See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, positive): “Then, we compare our IPVR with other methods on knowledge-based visual reasoning benchmarks [49,53] to demonstrate our method’s effectiveness and interpretability.”
- DoRA: Weight-Decomposed Low-Rank Adaptation (2024, methodology): “The training datasets contain several datasets from VQA (Goyal et al., 2017; Hudson & Manning, 2019; Marino et al., 2019; Schwenk et al., 2022), OCR (Mishra et al., 2019; Sidorov et al., 2020), region-level VQA (Kazemzadeh et al., 2014; Krishna et al., 2017; Mao et al., 2016), visual conversation…”
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, positive): “Our latency test experiment on A-OKVQA showed that LLaVA-13B model with FastV could achieve a lower latency than LLaVA-7B model while maintaining superior performance.”
- MultiModal-GPT: A Vision and Language Model for Dialogue with Humans (2023, methodology): “We utilize a diverse selection of vision and language instruction-following datasets in our study, including LLaVA [8], Mini-GPT4 [18], A-OKVQA [14], COCO Caption [7], and OCR VQA [10].”
- Aligning Large Multimodal Models with Factually Augmented RLHF (2023, methodology): “The basic pipeline of our multimodal RLHF can be summarized into three stages: Multimodal Supervised Fine-Tuning A vision encoder and a pre-trained LLM are jointly fine-tuned on an instruction-following demonstration dataset using token-level supervision to produce a supervised fine-tuned (SFT)…”