VQA: Visual Question Answering
By Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, Dhruv Batra. 2015
6,567 citations recorded by the source corpus.
We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ∼\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim $$\end{document}0.25 M images, ∼\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim $$\end{document}0.76 M questions, and ∼\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim $$\end{document}10 M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Yejin Choi
- Quoc V. Le
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Multimodal Learning
Citations in the corpus
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Striking examples include face recognition (Taigman et al., 2014) and visual question answering (Antol et al., 2015), where image features trained on ImageNet (Deng et al., 2009) and word embeddings trained on large unsupervised corpora are combined.”
- Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “…the advancement has been the use of neural attention mechanism, which enables the system to focus on a targeted area within a context paragraph (for MC) or within an image (for Visual QA), that is most relevant to answer the question (Weston et al., 2015; Antol et al., 2015; Xiong et al., 2016a).”
- Experience Grounds Language (2020, background): “…last decade of work descendant from image captioning (Farhadi et al., 2010; Mitchell et al., 2012), a myriad of tasks on visual question answering (Antol et al., 2015; Das et al., 2018), natural language and visual reasoning (Suhr et al., 2019b), visual commonsense (Zellers et al., 2019a), and…”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “For instance, in the first visual question answering dataset (Antol et al., 2015), biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether (Jabri et al., 2016; Goyal et al.,…”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “ext, although such representations are usually tailored for specic tasks. For example, MCB [9], BAN [16] and DFAF [10] proposed advanced multimodal fusion methods for Visual Question Answering (VQA) [3]. SCAN [20] and MAttNet [52] ? Equal contribution. 1 Code is available at https://github.com/ChenR”
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “ound the resulting dataset is class imbalanced, with 80% “yes” answers. Yes/No QA has been used in other contexts, such as the templated bAbI stories (Weston et al., 2015) or some Visual QA datasets (Antol et al., 2015; Wu et al., 2017). We focus on answering yes/no questions using natural language ”
- CoQA: A Conversational Question Answering Challenge (2018, background): “Like passages, other human-friendly sources are images and videos (Antol et al., 2015; Das et al., 2017; Hori et al., 2018).”
- Annotation Artifacts in Natural Language Inference Data (2018, background): “(2017) all showed that state-of-the-art visual question answering (Antol et al., 2015) systems leverage annotation biases in the dataset.”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “ble to discuss these other senses – for example, the contents of an image or a video. Work in this area encompasses image captioning (Lin et al., 2014), video captioning (Yu et al., 2016), visual QA (Antol et al., 2015), and more conversationally, visual dialogue (Das et al., 2017). Embodied agents ”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “…present (Rus-sakovsky et al., 2015; Lin et al., 2014), vision-and-language tasks, such as captioning (Chen et al., 2015), visual question answering (Antol et al., 2015), and visual reasoning (Suhr et al., 2019; Zellers et al., 2019), challenge systems to understand a wide range of detailed…”
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “We train and evaluate on the VQA 2.0 dataset [3] consisting of 1.1 million questions about COCO images [5] each with 10 answers.”
- Relation Networks for Object Detection (2017, background): “o not see any reason preventing it from finding broader applications in vision tasks, such as instance segmentation [35], action recognition [48], object relationship detection [33], caption [58], VQA [1], etc. Code is available at https://github.com/msracver/ Relation-Networks-for-Object-Detection. ”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “The model is pretrained on images from COCO and Visual Genome [17] as well as the image question answering datasets – VQA v2.”
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “, image-text retrieval [60], visual question answering [6], visual grounding [28, 58], etc.”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “VQA 2.0 dataset [11] contains 204k images and 1.1M questions about these images.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “ to signicant advances on a wide variety of computer vision problems such as object detection [7], semantic [8] and instance [9] segmentation, image captioning [10{12], and visual question answering [13,14]. Despite its practical success, this approach is expensive to scale since the pretraining ste”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “When finetuned on downstream tasks, these pre-trained models have achieved state-of-the-art performance across diverse V+L tasks, such as Visual Question Answering (VQA) [4, 17], Visual Commonsense Reasoning (VCR) [81], and Referring Expression Comprehension [78].”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”