Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions

By Arijit Ray, Gordon A. Christie, Mohit Bansal, Dhruv Batra, Devi Parikh. 2016

61 citations recorded by the source corpus.

Visual Question Answering (VQA) is the task of answering natural-language questions about images. We introduce the novel problem of determining the relevance of questions to images in VQA. Current VQA models do not reason about whether a question is even related to the given image (e.g. What is the capital of Argentina?) or if it requires information from external resources to answer correctly. This can break the continuity of a dialogue in human-machine interaction. Our approaches for determining relevance are composed of two stages. Given an image and a question, (1) we first determine whether the question is visual or not, (2) if visual, we determine whether the question is relevant to the given image or not. Our approaches, based on LSTM-RNNs, VQA model uncertainty, and caption-question similarity, are able to outperform strong baselines on both relevance tasks. We also present human studies showing that VQA models augmented with such question relevance reasoning are perceived as more intelligent, reasonable, and human-like.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

Citations in the corpus

Visual Dialog (2016, background): “See [40] for a related, but complementary effort on question relevance in VQA.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “ould make for a confusing and non-meaningful explanation to show to a user, if the goal is to convince them that the model has understood the question. One could add a component of question relevance [30] to identify better counter-examples. Instead, we take advantage of our balanced data collection”
Do explanations make VQA models more predictable to a human? (2018, background): “Ray et al. (2016) found that randomly pairing an image with a question in the VQA-1.”
Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation (2017, background): “, 2015) and in question answering on images (Antol et al., 2015; Ray et al., 2016; Malinowski and Fritz, 2014), when trained on large datasets, such as the COCO dataset (Lin et al.”
A Corpus of Natural Language for Visual Reasoning (2017, background): “VQA (Antol et al., 2015; Zitnick and Parikh, 2013) includes crowdsourced questions and answers for photographs and abstract scenes, and has been studied extensively (e.g., Lu et al., 2016; Xu and Saenko, 2016; Zhou et al., 2015; Chen et al., 2015a; Andreas et al., 2016b,a; Ray et al., 2016).”
Learning by Asking Questions (2017, background): “Motivated by the well-studied difficulty of analyzing results on real-world VQA datasets [22, 41, 57], Johnson et al.”
It Takes Two to Tango: Towards Theory of AI's Mind (2017, background): “Ray et al. [46] 6To control for familiarity, we ensured that subjects who perform a KP task are not allowed to perform an FP task.”
The Promise of Premise: Harnessing Question Premises in Visual Question Answering (2017, background): “We contrast our approach to the VTFQ dataset of Ray et al. (2016).”
Active Learning for Visual Question Answering: An Empirical Study (2017, background): “Note that such a model would need to not only reason about the informativeness of a question-image pair, but also about the relevance of a question to the image [27, 22].”
Question Part Relevance and Editing for Cooperative and Context-Aware VQA (C2VQA) (2017, methodology): “We experimented with the approach of [9] in using a 0,1, or 2 to denote the presence of a word in either the caption or question or both, but found raw word frequencies to perform better.”