Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

By Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi. 2017

723 citations recorded by the source corpus.

A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model - Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “Instead, the dominant strategy is to start with separate language and vision models pretrained for other large-scale tasks and then learn grounding as part of task training – often resulting in myopic groundings that generalize poorly when paired visiolinguistic data is limited or biased [9, 10].”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “While manual dataset creation is expensive and complex (Schwartz et al., 2017; Agrawal et al., 2018; Tsuchiya, 2018; Bras et al., 2020), crowdsourcing is the most popular method for goal-oriented, high quality/coverage datasets.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “Another supporting evidence is the issue of strong language priors (well-known in the VQA community), which led to interest in adversarial test sets and other methods to overcome strong language biases [1, 30, 14, 7].”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, background): “2019), new datasets or models to reduce the dataset bias (Zhang et al. 2016; Goyal et al. 2017; Agrawal et al. 2017) and ground the answer in the question (Lewis and Fan 2019).”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “As shown in [2], such models are less prone to exploiting the priors in the dataset.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “[2] partition the questions into training and validation sets such that their respective answer distributions become intentionally dissimilar.”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, background): “, Agrawal et al. (2018) propose a greedy split algorithm to maximize the coverage of test concepts in the train set while keeping question-type/answer pairs disjoint and observe performance degradation of existing approaches.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, methodology): “Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, methodology): “We compute a combined representation by copying the question representation to every spatial location in the CNN, and concatenating it with (or simply adding it to) the visual features, like previous Otherwise, we follow the canonical Visual QA pipeline [7,9,22,23,24,25].”
Scene Text Visual Question Answering (2019, background): “Since its introduction, VQA has received a lot of attention from the Computer Vision community [4, 11, 46, 16, 23, 2] facilitated by access to large-scale datasets that allow the training of VQA models [4, 16, 33, 58, 52, 40].”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “We didn’t use proxy vision-language tasks like captioning [65, 67] and VQA [54, 15] as the diagnosis, because their implementations have too many components unrelated to SGG and their datasets are challenged by their own biases [1, 14, 34].”
Learning by Abstraction: The Neural State Machine (2019, methodology): “set biases and superﬁcial statistics [1, 44, 27], and are known to be particularly difﬁcult for neural networks [51]. Here, we follow the standard VQA1/2 [27] accuracy metric for this task (deﬁned in [3]). Table 3 presents our performance compared to existing approaches. We can see that NSM surpasse”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “ and language that are notoriously subject to poor generalization due to dataset biases. Our experiments aim (1) to measure the impact of gradient supervision on performance for well-known tasks, and (2) to demonstrate that the necessary annotations are available in a variety of existing datasets. W”
There is Strength in Numbers: Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020, background): “In visual question answering datasets, several studies found it is often possible to answer the question without looking at the corresponding image (Zhang et al., 2016; Kaﬂe and Kanan, 2016; Goyal et al., 2017; Agrawal et al., 2018).”
Compact Trilinear Interaction for Visual Question Answering (2019, background): “However, answer information in FFOE VQA [1, 40, 26, 39] is only available in the training phase but not in the testing phase.”
Fooling Vision and Language Models Despite Localization and Attention Mechanism (2017, background): “that the VQA models capture the training data biases and tend to predict the most frequent answers [1, 20, 27, 31].”