An Analysis of Visual Question Answering Algorithms

By Kushal Kafle, Christopher Kanan. 2017

264 citations recorded by the source corpus.

In visual question answering (VQA), an algorithm must answer text-based questions about images. While multiple datasets for VQA have been created since late 2014, they all have flaws in both their content and the way algorithms are evaluated on them. As a result, evaluation scores are inflated and predominantly determined by answering easier questions, making it difficult to compare different methods. In this paper, we analyze existing VQA algorithms using a new dataset called the Task Driven Image Understanding Challenge (TDIUC), which has over 1.6 million questions organized into 12 different categories. We also introduce questions that are meaningless for a given image to force a VQA system to reason about image content. We propose new evaluation schemes that compensate for over-represented question-types and make it easier to study the strengths and weaknesses of algorithms. We analyze the performance of both baseline and state-of-the-art VQA models, including multi-modal compact bilinear pooling (MCB), neural module networks, and recurrent answering units. Our experiments establish how attention helps certain categories more than others, determine which models work better than others, and explain how simple models (e.g. MLP) can surpass more complex models (MCB) by simply learning to answer large, easy question categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

A simple neural network module for relational reasoning (2017, neutral): “The results so far led us to hypothesize that the difficulty in solving CLEVR lies in its heavy emphasis on relational reasoning, contrary to previous claims that the difficulty lies in question parsing [9].”
VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “0: Real [17] MSCOCO [25] Crowd workers (AMT) Typed Visual7W [45] MSCOCO [25] Crowd workers (AMT) Typed CLEVR [19] Synthetic Shapes Automatically generated (templates) ——– SHAPES [6] Synthetic Shapes Automatically generated (templates) ——– Visual Genome [23] MSCOCO [25] & YFCC100M [35] Crowd workers ”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “In terms of accuracy metric, [25] proposed various accuracy metrics for VQA to analyze performance of VQA models on rare answers.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [16, 29, 30, 15].”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “For our experiments, we use the TDIUC [56] and CLEVR [57] VQA datasets.”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “More recently, [18] propose two new evaluation metrics that compensate for the skewed distribution of question types and for the skewed distribution of answers within each question type in the test set.”
Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, background): “Most of the literature relies on data groups that are very coarse and easy to program (e.g., based on question length and answer types (Kaﬂe and Kanan, 2017; Agrawal et al., 2016; Shen et al., 2017)).”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [18, 35, 36, 17].”
Did the Model Understand the Question? (2018, background): “Kaﬂe and Kanan (2017); Zhang et al. (2016) examine the VQA data, identify de-ﬁciencies, and propose data augmentation to reduce over-representation of certain question/answer types.”
VQD: Visual Query Detection In Natural Scenes (2019, methodology): “The questions are generated using multiple templates for each question type, which is an approach that has been used in earlier work for VQA (Kafle and Kanan, 2017a; Kafle et al., 2017).”
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization (2017, background): “One particular kind of structured layer is concerned with global covariance pooling after the last convolution layer, which has shown impressive improvement over the classical firstorder pooling, successfully used in FGVC [25], visual question answering [15] and video action recognition [34].”
FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding (2020, background): “TDIUC [37] Real-world images No No varies in size 7 7 X”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “The rapid development of VQA has been supported by the emergence of various benchmark datasets [1, 3, 14, 28, 29, 32, 52] and methods [13, 59, 66, 72, 54, 46, 77, 78, 79, 80].”
Overcoming Language Priors in Visual Question Answering with Adversarial Regularization (2018, background): “In response to these challenges, there has been extensive work on VQA in recent years both in terms of dataset curation [6, 12, 2, 17, 13, 32, 3] and modeling [2, 5, 28, 14, 16, 4, 22, 20].”
All You May Need for VQA are Image Captions (2022, background): “In addition, the brittleness of VQA systems trained on such manual annotations is well-understood and documented (Agrawal et al., 2018; Kaﬂe and Kanan, 2017).”
Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “Second, the questioner tends to ask about the objects seen in the image, which leads to the “visual priming bias” [8, 19, 25].”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “Table 2 presents detail performances with Acc metric over each question category of TDIUC when all categories, including “Absurd”, are used for training.”
Multi-Modality Latent Interaction Network for Visual Question Answering (2019, background): “0 [12] and TDIUC [20] benchmarks.”