Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

By Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yibing Liu, Yinglong Wang, Mohan Kankanhalli. 2019

37 citations recorded by the source corpus.

Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years. Although some progress has been achieved so far, several studies have pointed out that current VQA models are heavily affected by the language prior problem, which means they tend to answer questions based on the co-occurrence patterns of question keywords (e.g., how many) and answers (e.g., 2) instead of understanding images and questions. Existing methods attempt to solve this problem by either balancing the biased datasets or forcing models to better understand images. However, only marginal effects and even performance deterioration are observed for the first and second solution, respectively. In addition, another important issue is the lack of measurement to quantitatively measure the extent of the language prior effect, which severely hinders the advancement of related techniques. In this paper, we make contributions to solve the above problems from two perspectives. Firstly, we design a metric to quantitatively measure the language prior effect of VQA models. The proposed metric has been demonstrated to be effective in our empirical studies. Secondly, we propose a regularization method (i.e., score regularization module) to enhance current VQA models by alleviating the language prior problem as well as boosting the backbone model performance. The proposed score regularization module adopts a pair-wise learning strategy, which makes the VQA models answer the question based on the reasoning of the image (upon this question) instead of basing on question-answer patterns observed in the biased training set. The score regularization module is flexible to be integrated into various VQA models. We conducted extensive experiments over two popular VQA datasets (i.e., VQA 1.0 and VQA 2.0) and integrated the score regularization module into three state-of-the-art VQA models. Experimental results show that the score regularization module can not only effectively reduce the language prior problem of these VQA models but also consistently improve their question answering accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Achieving Human Parity on Visual Question Answering (2021, methodology): “For example, [23, 44] address the Language-Prior problem in VQA and propose a reweighting-basedmethod to reason based on both the image and text content.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “Several models and training meth-ods [11,15,16,26,27,40,52] have been proposed with signiﬁcant improvements.”
LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering (2021, background): “However, recent studies [2, 16, 25] demonstrate that most existing VQA models overly leverage superficial correlations between question and answer words to generate answers, i.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, background): “Specially, the rank-3 sentence of the third image query in Figure 8 demonstrates that it is still challenging to count entities exactly due to the language prior problem [9] on existing datasets.”
VQA With No Questions-Answers Training (2018, background): “The language prior was addressed as well [22, 20, 13, 76, 57].”