The Promise of Premise: Harnessing Question Premises in Visual Question Answering

By Aroma Mahendru, Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, Stefan Lee. 2017

39 citations recorded by the source corpus.

In this paper, we make a simple observation that questions about images often contain premises – objects and relationships implied by the question – and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.

Subfields: Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “• Q+C: the question and caption are encoded by two separate LSTMs and then the features of the question and caption are concatenated and input to a softmax layer.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, methodology): “Contrary to GQA, those datasets are either small in scale [22] or use only a restricted set of objects and a handful of non-compositional templates [15, 21].”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “Somewhat related to our work are several datasets created through templates [25, 17, 24], most of them for the purpose of data augmentation.”
Data augmentation techniques for the Video Question Answering task (2020, background): “loiting purely linguistic aspects or by using visual information to better guide the generation.Yet, as the authors report in the respective papers, these methods although reliable are not error-free [22,14]; on the other hand, our proposed techniques are simple yet eective and they do not raise iss”
Active Learning for Visual Question Answering: An Empirical Study (2017, background): “Note that such a model would need to not only reason about the informativeness of a question-image pair, but also about the relevance of a question to the image [27, 22].”