RUBi: Reducing Unimodal Biases in Visual Question Answering
By Rémi Cadène, Corentin Dancette, Hedi Ben-Younes, Matthieu Cord, Devi Parikh. 2019
442 citations recorded by the source corpus.
Visual Question Answering (VQA) is the task of answering questions about an image. Some VQA models often exploit unimodal biases to provide the correct answer without using the image information. As a result, they suffer from a huge drop in performance when evaluated on data outside their training set distribution. This critical issue makes them unsuitable for real-world settings. We propose RUBi, a new learning strategy to reduce biases in any VQA model. It reduces the importance of the most biased examples, i.e. examples that can be correctly classified without looking at the image. It implicitly forces the VQA model to use the two input modalities instead of relying on statistical regularities between the question and the answer. We leverage a question-only model that captures the language biases by identifying when these unwanted regularities are used. It prevents the base VQA model from learning them by influencing its predictions. This leads to dynamically adjusting the loss in order to compensate for biases. We validate our contributions by surpassing the current state-of-the-art results on VQA-CP v2. This dataset is specifically designed to assess the robustness of VQA models when exposed to different question biases at test time than what was seen during training. Our code is available: this http URL
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Zhe Gan
- Percy Liang
- Yejin Choi
- Luke S. Zettlemoyer
- Yoshua Bengio
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Bernt Schiele
- Cordelia Schmid
- Ming-Hsuan Yang
- Andrew Zisserman
In Multimodal Learning
- Zhe Gan
- Linjie Li
- Hanwang Zhang
- Jingjing Liu
- Anton van den Hengel
- Damien Teney
- Jianfeng Gao
- Jianfei Cai
In Question Answering & Reading Comprehension
- Damien Teney
- Anton van den Hengel
- Xiaodong He
- Yonatan Belinkov
- Hanwang Zhang
- Radu Soricut
- Ehsan Abbasnejad
- Piyush Sharma
Citations in the corpus
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “Another supporting evidence is the issue of strong language priors (well-known in the VQA community), which led to interest in adversarial test sets and other methods to overcome strong language biases [1, 30, 14, 7].”
- Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “Several models and training meth-ods [11,15,16,26,27,40,52] have been proposed with significant improvements.”
- A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “2In some literature [68, 17, 31], this is referred to as out-of-distribution generalization; we follow [8, 72], and term it as robustness to reduce confusion and unify the terminology.”
- LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, methodology): “4) Non-annotation based methods, including AdvReg [25], GRL [15], RUBi [8], VGQE [13], DLR [18] and LMH [10].”
- Learning from others' mistakes: Avoiding dataset biases without modeling them (2020, background): “, 2019) and dynamic loss adjustment (Cadène et al., 2019).”
- Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “RUBi [10] Baseline 44.”
- Deconfounded Image Captioning: A Causal Retrospect (2020, background): “…by observing that our biological vision-language system works well even though the around real word is biased, we are prompted to evolve from diagnosis to treatment, by changing from collecting new datasets [3], [10], [11], [12] to designing unbiased models [1], [13] for confronting the bias.”
- Learning De-biased Representations with Biased Representations (2019, methodology): “The prior methods include RUBi (Cadene et al., 2019) and LearnedMixin+H (Clark et al.”
- On Modality Bias in the TVQA Dataset (2020, unknown): “The RUBi (Reducing Unimodal Bias) learning strategy [9] uses a question-only model to apply a learnable mask on the outputs of the full model during training.”
- Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “Other methods have been proposed to handle biased priors using adversarial training or additional training-only structures [248], [249].”
- Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, methodology): “Second, we provide a novel causality-based interpretation for recent debiasing VQA works [10, 13].”
- REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, methodology): “model biases, but specifically formodels trained in theVisual Question Answering (VQA) domain (Cadene et al., 2019).”
- VQA With No Questions-Answers Training (2018, background): “The language prior was addressed as well [22, 20, 13, 76, 57].”