Visual question answering: Datasets, algorithms, and future challenges

By Kushal Kafle, Christopher Kanan. 2016

268 citations recorded by the source corpus.

Abstract Visual Question Answering (VQA) is a recent problem in computer vision and natural language processing that has garnered a large amount of interest from the deep learning, computer vision, and natural language processing communities. In VQA, an algorithm needs to answer text-based questions about images. Since the release of the first VQA dataset in 2014, additional datasets have been released and many algorithms have been proposed. In this review, we critically examine the current state of VQA in terms of problem formulation, existing datasets, evaluation metrics, and algorithms. In particular, we discuss the limitations of current datasets with regard to their ability to properly train and assess VQA algorithms. We then exhaustively review existing algorithms for VQA. Finally, we discuss possible future directions for VQA and image understanding research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “This phenomenon has been observed in image captioning [9] as well as visual question answering [59,61,21,24, 1].”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [16, 29, 30, 15].”
Inferring and Executing Programs for Visual Reasoning (2017, background): “ring, reasoning-augmented models, semantic parsers, and (neural) program-induction methods. Visual question answering (VQA) is a popular proxy task for gauging the quality of visual reasoning systems [21,44]. Like the CLEVR dataset, benchmark datasets for VQA typically comprise a set of questions on”
Measuring Catastrophic Forgetting in Neural Networks (2017, background): “These algorithms power most of the recent advances in semantic segmentation (Long, Shelhamer, and Darrell 2015), visual question answering (Kafle and Kanan 2017), and reinforcement learning (Mnih et al. 2013).”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [18, 35, 36, 17].”
Active Learning for Visual Question Answering: An Empirical Study (2017, background): “Such trends are invariant to choice of image feature [15] and is also observed in ImageNet image classification [24].”
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering (2019, methodology): “Although these methods have achieved state-of-the-art performance over their contemporary baselines, many studies point out that today’s VQA models are yet heavily driven by superficial correlations between questions and answers in the training data and lack of sufficient visual understanding [2, 18”
VQD: Visual Query Detection In Natural Scenes (2019, methodology): “VQD is related to the visual referring expression recognition (RER) task (Kazemzadeh et al., 2014); however, in RER every image has only a single correct box.”
Learning Conditioned Graph Structures for Interpretable Visual Question Answering (2018, background): “"Would you want to fly in that plane?") [23].”
Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, background): “Some problems requiring these additional components include visual question answering [8, 44, 56], referring expression recognition [45, 68], visual query detection [3], and image captioning [11].”
Structured Attentions for Visual Question Answering (2017, methodology): “The method was surpassed by some simple baseline models due to its demand for better semantic segmentation [15].”
Learning Answer Embeddings for Visual Question Answering (2018, background): “Though such methods naturally limited themselves to answer questions within a fixed (usually small) vocabulary, this framework has been shown to outperform other methods that dedicate for freeform answer generation [25, 14].”