VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions

By Qing Li, Qingyi Tao, Shafiq R. Joty, Jianfei Cai, Jiebo Luo. 2018

123 citations recorded by the source corpus.

Most existing works in visual question answering (VQA) are dedicated to improving the accuracy of predicted answers, while disregarding the explanations. We argue that the explanation for an answer is of the same or even more importance compared with the answer itself, since it makes the question and answering process more understandable and traceable. To this end, we propose a new task of VQA-E (VQA with Explanation), where the computational models are required to generate an explanation with the predicted answer. We first construct a new dataset, and then frame the VQA-E problem in a multi-task learning architecture. Our VQA-E dataset is automatically derived from the VQA v2 dataset by intelligently exploiting the available captions. We have conducted a user study to validate the quality of explanations synthesized by our method. We quantitatively show that the additional supervision from explanations can not only produce insightful textual sentences to justify the answers, but also improve the performance of answer prediction. Our model outperforms the state-of-the-art methods by a clear margin on the VQA v2 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “Alternatively, we perform answer rationale prediction (Li et al. 2018; Park et al. 2018; Zellers et al. 2019) in a zero-shot manner, and generate answer rationale as an open-ended text generation task.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, neutral): “[19].”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, background): “VQAE [29] is another dataset that also builds on top of VQA v2.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, methodology): “Similarly, the VQA-E dataset [75] converts image captions from the VQA V2.”
On the Diversity and Limits of Human Explanations (2021, background): “We cannot infer the exact question Li et al. (2018) visual QA automatically generated, We cannot infer the exact question Park et al. (2018) visual QA During data annotation, we ask the annotators to complete the sentence “I can tell the person is doing (action) because..” where the action is the…”
VQA With No Questions-Answers Training (2018, methodology): “Some methods provide reasoning by using ”facts” extraction ( e.g . scene type) [39] or image caption results [28, 1, 27].”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “Both E-SNLI-VE and VQA-E present novel and cost-effective ways for producing large E X NLP datasets for new tasks.”
Challenges and Prospects in Vision and Language Research (2019, background): “Several works propose the use of textual explanations to improve interpretability (Hendricks et al., 2016; Li et al., 2018).”
Teaching Machine Comprehension with Compositional Explanations (2020, background): “Li et al. (2018) formulated visual question answering as multi-task learning by requiring the model to generate an explanation based on the hidden representation constructed from image and question.”
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020, methodology): “The bottom-up top-down attention (BUTD) model (Anderson et al., 2018) has been proposed to incorporate rationales with visual features for VQA-E and E-SNLI-VE (Li et al., 2018; Do et al., 2020).”