Counterfactual VQA: A Cause-Effect Look at Language Bias

By Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xiansheng Hua, Ji-rong Wen. 2020

539 citations recorded by the source corpus.

VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “There are very few recent works [36, 24, 37, 42, 52, 13, 66] trying to endow the model with the capability of causal reasoning.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “This challenge not only reveals fundamental issues of LVLMs, such as over-reliance on statistical bias [1, 2, 19, 38] and unimodal priors [21, 22, 51, 68, 70, 75], but also has direct implications for the practical deployment of LVLMs.”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “Recently, there is a growing number of computer vision tasks that benefit from causality [34, 40, 51, 52, 56, 63, 65].”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “On the other hand, there is an increasing demand for the multimodality of knowledge to break through the bottleneck of real world applications [16], [17], [18].”
LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, methodology): “2) ensemblebased [8, 10, 19, 24]: they use the ensemble strategy to combine the two models’ prediction, which derives the training gradients based on the fused answer distributions.”
Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “Motivated by the success of knowledge distillation [33, 13] and cross-domain knowledge [30, 31, 55] in knowledge transfer, we leverage the zero-shot CLIP predictions as the general knowledge, and compare the fine-tuned predictions with the general knowledge to regularize the gradient direction.”
Deconfounded Image Captioning: A Causal Retrospect (2020, methodology): “For [62], [65], they apply other causality techniques: Natural Indirect Effect and Total Direct Effect for eliminating the mediator effect instead of deconfounding.”
Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System (2020, background): “to be exploited in computer vision domain such as scene graph generation [12, 41], visual explanations [29] and vision-language multi-modal learning [30, 33, 42].”
What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (2023, background): “[18] combines the idea of counterfactual reasoning with computer vision.”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, background): “Another line of research performs counterfactual inference over the predictions of deep model to incorporate counterfactual thinking (Yue et al., 2021; Wang et al., 2021; Niu et al., 2021; Tang et al., 2020).”
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021, background): “to hallucinate the inconsistency tokens via introducing spurious linguistic correlations or even biased knowledge learned from the training corpus (Hermann et al., 2015; Niu et al., 2020).”
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect (2020, unknown): “Recently, causal inference has also attracted increasing attention in computer vision society [39–44] for removing the dataset bias in domain-specific applications, e.g. , using pure direct effect to capture the spurious bias in VQA [41] and NWGM for Captioning [42].”
Conterfactual Generative Zero-Shot Semantic Segmentation (2021, methodology): “…spurious correlation in few-shot learning and image annotation ZS3Net Our model image caption [19, 20]; direct and indirect effect help to generate unbiased scene graph [21]; counterfactual causality was widely used in visual question answer (VQA) problem and long-tailed classiﬁcation [22, 23].”
Grounding Answers for Visual Questions Asked by Visually Impaired People (2022, background): “A key challenge the community has faced in developing such datasets is the language bias problem [12, 23, 27, 30].”
A Survey on Causal Inference (2020, background): “In computer vision, some emerging tasks that lie in the interaction between vision and language, such as visual question answering (VQA) and image captioning, have become a fundamental block for many frontier interactive AI systems [100].”