Towards Transparent AI Systems: Interpreting Visual Question Answering Models

By Yash Goyal, Akrit Mohapatra, Devi Parikh, Dhruv Batra. 2016

76 citations recorded by the source corpus.

Deep neural networks have shown striking progress and obtained state-of-the-art results in many AI research fields in the recent years. However, it is often unsatisfying to not know why they predict what they do. In this paper, we address the problem of interpreting Visual Question Answering (VQA) models. Specifically, we are interested in finding what part of the input (pixels in images or words in questions) the VQA model focuses on while answering the question. To tackle this problem, we use two visualization techniques -- guided backpropagation and occlusion -- to find important words in the question and important regions in the image. We then present qualitative and quantitative analyses of these importance maps. We found that even without explicit attention mechanisms, VQA models may sometimes be implicitly attending to relevant regions in the image, and often to appropriate words in the question.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Question Answering & Reading Comprehension

Citations in the corpus

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, methodology): “…have focused on visualizing, comparing, and understanding the operation of deep learning networks, with an audience of other researchers (e.g., Goyal et al., 2016abc; Karpathy et al., 2015; Kim, et al., 2016; Moeyersoms, et. al., 2016; Sadeghi, et al., 2015; Selvaraju, et al., 2017; Yosinski,…”
Stacking with Auxiliary Features for Visual Question Answering (2018, background): “Visual Question Answering (VQA), the task of addressing open-ended questions about images (Ma-linowski and Fritz, 2014; Antol et al., 2015), has attracted signiﬁcant attention in recent years (Andreas et al., 2016a; Goyal et al., 2016; Agrawal et al., 2016; Teney et al., 2017).”
Interpretable Convolutional Neural Networks (2017, methodology): “[6] used a gradient-based method to interpret visual questionanswering.”
Visual interpretability for deep learning: a survey (2018, methodology): “[Wang et al., 2017; Goyal et al., 2016] have tried to interpret the logic for visual question-answering encoded in neural networks.”
Examining CNN representations with respect to Dataset Bias (2017, methodology): “(Goyal et al. 2016) used a gradient-based method to interpret visual question-answering models.”
Do explanations make VQA models more predictable to a human? (2018, background): “Additionally, the VQA 1.0 dataset contains label biases; therefore, the model is very likely to answer “white’ to a “what color” question (Goyal et al., 2016a).”
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (2016, background): “ are related to the prediction. Other models [47, 14, 48] aim to understand which intermediate features are important for a classiﬁcation decision, e.g. what does a certain neuron represent. Recently [17] proposed to tackle the problem of indicating the evidence of a prediction through guided backpr”
FOIL it! Find One mismatch between Image and Language caption (2017, methodology): “Following (Goyal et al., 2016b), we systematically occlude subsets of the language input, forward propagate the masked input through the model, and compute the change in the probability of the answer predicted with the unmasked original input.”
Explainable Artificial Intelligence: a Systematic Review (2020, background): “[331] used the same evaluation approach of [329, 330] to compare LRP with Occlusion Sensitivity and Sensitivity Analysis.”
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers (2018, unknown): “Another popular technique for analyzing particular data instances using text data instead of images is to color the background of particular characters in a phrase of words in a sentence using a divergent color scheme according to some criteria, oftentimes their activation magnitude [32], [36], [40]”
Using Explanations to Improve Ensembling of Visual Question Answering Systems (2017, methodology): “VQA addresses open-ended questions about images [Antol et al. , 2015] and has attracted signiﬁcant attention in the past year [Andreas et al. , 2016; Goyal et al. , 2016; Agrawal et al. , 2016].”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “s on the balanced set, and ﬁnally the novel VQA model with counter-example based explanations. Models with explanation. A number of recent works have proposed mechanisms for generating ‘explanations’ [12,33,47,11,31] for the predictions made by deep learning models, which are typically ‘black-box’ a”
It Takes Two to Tango: Towards Theory of AI's Mind (2017, result): “Several works propose introspective explanations based on internal states of a decision process [67, 53, 26, 69], while others generate justifications consistent with model outputs [50, 29, 43].”