Overcoming Language Priors in Visual Question Answering with Adversarial Regularization

By Sainandan Ramakrishnan, Aishwarya Agrawal, Stefan Lee. 2018

272 citations recorded by the source corpus.

Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training such as overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -- discouraging the VQA model from capturing language biases in its question encoding. Further,we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -- achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Adversarial Datasets Past work has proposed the idea of creating adversarial datasets, whether by balancing the dataset with respect to priors [25, 28, 62] or switching them at test time [1].”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, methodology): “Instead we follow prior work (Grand and Belinkov, 2019; Ramakrishnan et al., 2018) and perform model selection on the test set.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “Other All YesNo Nb Other All YesNo Nb Other SAN [76] { { { { 25.0 38.4 11.1 21.7 { { { { GVQA [2] { { { { 31.3 58.0 13.7 22.1 { { { { UpDown [64] { { { { 39.1 62.4 15.1 34.5 Ramakrishnan et al., 2018 [52] { { { { 42.0 65.5 15.9 36.6 { { { { Grand and Belinkov, 2019 [26] { { { { 42.3 59.7 14.8 40.8 {”
Answer Them All! Toward Universal Visual Question Answering Models (2019, unknown): “GVQA [3] and UpDn with adversarial regularization [45].”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “So far, the most effective debiasing models for VQA are ensemble-based meth-ods [33, 18, 7, 10, 14, 27].”
Learning Credible Deep Neural Networks with Rationale Regularization (2019, background): “In order to alleviate the influence of unwanted dataset bias to models’ performance, one line of work tackles this problem by regulating the training of models [22], [23], while some others consider to construct challenging datasets by eliminating biases and annotation artifacts [11], [24].”
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering (2019, methodology): “12 - - - - - - - SAN+Q-Adv+DoE [32] 71.”
On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference (2019, background): “…biases, as well as similar biases discovered in other natural language understanding tasks (Schwartz et al., 2017), including visual question answering, where recent work has considered similar adversarial techniques for removing language biases (Ramakrishnan et al., 2018; Grand & Belinkov, 2019).”
RUBi: Reducing Unimodal Biases in Visual Question Answering (2019, methodology): “The work most related to ours in terms of approach is [25].”
Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, methodology): “[21], RUBi [1] and Learned-Mixin (LM) [2].”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, background): “…paradigms, such as adversarial training (Zhu et al., 2020; Fu et al., 2020; Teney et al., 2020), contrastive learning (Liang et al., 2020), causal graph (Gokhale et al., 2020), posterior regularization (Ramakrishnan et al., 2018), and designing new learning paradigms (Gokhale et al., 2020).”
Visual Commonsense R-CNN (2020, background): “Previous works blame this for dataset bias without further justification [24, 44, 53, 7], e.”
Grounding Answers for Visual Questions Asked by Visually Impaired People (2022, background): “A key challenge the community has faced in developing such datasets is the language bias problem [12, 23, 27, 30].”
On Modality Bias in the TVQA Dataset (2020, unknown): “Flexible model-agnostic learning strategies have recently been developed that are trained end-to-end [9, 11, 38].”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “[65] Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee.”