Visual question answering: A survey of methods and datasets

By Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony R. Dick, Anton van den Hengel. 2016

469 citations recorded by the source corpus.

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Yes/No QA has been used in other contexts, such as the templated bAbI stories (Weston et al., 2015) or some Visual QA datasets (Antol et al., 2015; Wu et al., 2017).”
Grounding Visual Explanations (2018, methodology): “IC-Wang [31] is an image captioning model whereas HieCoAtt [16] is an attention based VQA model.”
Open-Ended Long-form Video Question Answering via Adaptive Hierarchical Reinforced Networks (2018, methodology): “A survey of existing image question answering methods can be found in [Wu et al. , 2016].”
Zero-Shot Visual Question Answering (2016, methodology): “This distinction reﬂects the fact that CNNs pre-trained on ImageNet [4] are commonly used in existing VQA methods, and the fact that VQA is the task that we are actually interested in.”
FOIL it! Find One mismatch between Image and Language caption (2017, background): “There is a wealth of literature on both tasks, but we only discuss here the ones most related to our work and refer the reader to the recent surveys by (Bernardi et al., 2016; Wu et al., 2016).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “In this section we describe only the evaluation measures used for Image Question Answering as Models, Results, and some Discussion are extensively presented in the recent surveys (Wu et al., 2017).”
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering (2019, methodology): “Generally speaking, the existing VQA methods can be classiﬁed into four categories [37]: JointEmbedding , AttentionMechanism-based , Compositional and Knowledge Base-enhanced .”
Syntax-Aware Action Targeting for Video Captioning (2020, unknown): “It has extensive applications such as Visual Question Answering (VQA) [57, 16], human-robot interaction [32] and video retrieval [64, 14].”
The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions (2016, background): “The survey paper [34] makes a similar observation - “98.8% of the ‘when’ questions in the Visual Genome can not be directly answered from the provided scene graph annotations” .”
Graph-Structured Representations for Visual Question Answering (2016, background): “The task of Visual Question Answering has received growing interest in the recent years (see [17, 4, 25] for example).”
Data Augmentation for Visual Question Answering (2017, methodology): “Detailed reviews of existing methods can be found in Kaﬂe and Kanan (2017) and Wu et al. (2016).”
Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering (2018, methodology): “A survey article (Wu et al. 2016) on VQA dissects the different methods into the following categories: i) Joint Embed-ding methods, ii) Attention Mechanisms, iii) Compositional Models, and iv) Models using External Knowledge Bases.”
VQA With No Questions-Answers Training (2018, background): “Current approaches to handle VQA by a machine [67, 62, 55, 77, 32] take a different path, where most answering systems are trained directly to select an answer from common answers of a training set, based on fused image features (mostly using a pre-trained CNN [25]) and question features (mostly usi”