Visualizing Deep Neural Network Decisions: Prediction Difference Analysis

By Luisa M. Zintgraf, Taco Cohen, Tameem Adel, Max Welling. 2017

746 citations recorded by the source corpus.

This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “However, [23] consider somewhat simple data and generating explanations for more complex tasks (like image captioning) is a difficult and active area of research [24,25,26,27,28,29].”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “, 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al., 2016; Zintgraf et al., 2017), with many similarities among the methods (Ancona et al.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “sed Explanations (ACE) algorithm [55] uses super-pixels as explanations. Other decomposition methods that have been developed include, DeepLIFT and gradient*input [56], Prediction Difference Analysis [57] and [40]. Peak Response Mapping [58] is generated by backpropagating peak signals. Peak signals”
Sanity Checks for Saliency Maps (2018, methodology): “Aside gradient-based approaches, other methods ‘learn’ an explanation per sample for a model [20, 17, 12, 15, 11, 21].”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Attribution, a term introduced by Ancona et al. (2018), also referred to as relevance (Bach et al., 2015; Binder et al., 2016; Zintgraf et al., 2017; Robnik-ˇSikonja and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or inﬂuence (Kindermans et al.,…”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “Perhaps the most prevalent form of visual explanation rely on producing heat maps/attention maps which indicate which region of an image is most important for a decision [13, 29, 37, 41].”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, methodology): “While there are several different approaches for obtaining local-explanations (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Pieter-Jan Kindermans, 2018), recent work by Ancona et al. (2017) and Lundberg & Lee (2017) both show that there are equivalences…”
A Survey of Methods for Explaining Black Box Models (2018, methodology): “In Reference [143] the authors present a probabilistic methodology f for explaining classification decisions made by DNN.”
Challenges for Transparency (2017, background): “those parts which in a sense were most responsible for leading to the system’s prediction (Ribeiro et al., 2016; Zintgraf et al., 2017).”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, methodology): “Furthermore, since our approach is after training just as expensive as a single back-propagation step, it can be applied in a real-time context, which is also possible for the work done by Dabkowski & Gal (2017) but not for Zintgraf et al. (2017).”
Learning Important Features Through Propagating Activation Differences (2017, background): “Zintgraf et al. (Zintgraf et al., 2017) proposed a clever strategy for analyzing the difference in a prediction after marginalizing over each input patch.”
Improving Interpretability of Deep Neural Networks with Semantic Information (2017, methodology): “To analyze the correspondence between neurons and topics and semantically interpret the learned features, we pro-pose a prediction difference maximization algorithm, which is similar with a concurrent and independent work [41], to represent the learned features of each neuron by a topic.”
Visual interpretability for deep learning: a survey (2018, methodology): “Studies of [Zintgraf et al., 2017; Kindermans et al., 2017] have invented methods to visualize areas in the input image that contribute the most to the decision-making process of the CNN. [Wang et al., 2017; Goyal et al., 2016] have tried to interpret the logic for visual question-answering encoded…”
A unified view of gradient-based attribution methods for Deep Neural Networks (2017, background): “ While perturbation-based methods allow a direct estimation of the marginal effect of a feature, they tend to be very slow as the number of features to test grows (ie. up to hours for a single image (Zintgraf et al., 2017)). What is more, given the nonlinear nature of DNNs, the result is strongly in”
TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2018, background): “the input which were responsible for a particular prediction is considered a viable explanation in many scenarios including domains like self-driving cars [23], finance [9] and medical imaging [24].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “More recently, marginalizing over the plausible values of an input has been suggested as a way to more accurately estimate contributions [333].”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Recent works on effective receptive fields [30] and salient regions [40, 41, 12, 6] reveal that only a small proportion of pixels in the theoretical receptive field contribute significantly to the final network prediction.”