Evaluating the Visualization of What a Deep Neural Network Has Learned

By Wojciech Samek, Alexander Binder, Grgoire Montavon, Sebastian Lapuschkin, K. Müller. 2015

1,387 citations recorded by the source corpus.

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the “importance” of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

Axiomatic Attribution for Deep Networks (2017, methodology): “For instance, in the context of an object recognition task, (Samek et al., 2015) suggests that we select the top k pixels by attribution and randomly vary their intensities and then measure the drop in score.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “construct saliency maps for interpretability include [13], [45]– [50].”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…and Clune, 2015; Papernot et al., 2017; Rajani and Mooney, 2017; Reed, Akata, Lee, and Schiele, 2016; Sadeghi, Kumar Divvala, and Farhadi, 2015; Samek et al., 2017; Schank, 2013; Schmidhuber, 2015; Shwartz-Ziv and Tishby, 2017; Simonyan, Vedaldi, and Zisserman, 2013; Springenberg, Dosovitskiy,…”
Sanity Checks for Saliency Maps (2018, background): “[27] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.”
A Survey of Methods for Explaining Black Box Models (2018, methodology): “In [103] it is presented a comparison between explanations provided with LRP and more traditional deconvolution and sensitivity analysis.”
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, background): “The authors of [31] proposed such a quality measure based on perturbation analysis.”
Methods for interpreting and understanding deep neural networks (2017, methodology): “A quantitative framework allows to compare explanation techniques speciﬁcally for a target problem, e.g. ILSVRC or MIT Places [42].”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “gradient-based techniques (Samek et al., 2017), and were also successfully transferred to neural networks for text data (Arras et al.”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, methodology): “We measure how the prediction (after the soft-max) for the initially selected class changes as a function of corrupting more and more patches based on the ordering assigned by the attribution (see Samek et al., 2016).”
Explaining nonlinear classification decisions with deep Taylor decomposition (2015, methodology): “See [27] for a quantitative comparison of sensitivity analysis and relevance propagation methods similar to deep Taylor decomposition on this data.”
A unified view of gradient-based attribution methods for Deep Neural Networks (2017, methodology): “Our method differs from [15] by two aspects: 1) we always remove features based on their ranking and we do not assume our inputs are natural images, where local pixel coherence can be exploited to alter local patches; 2) while [15] performs this test removing most relevant features first (MoRF), we ”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “Some prior work has evaluated the applicability of saliency maps for explaining the behavior of image classiﬁers (Samek et al., 2017; Adebayo et al., 2018; Kindermans et al., 2019), but there is not a corresponding literature evaluating the applicability of saliency maps for explaining RL agent…”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, background): “An advantage of the decomposition framework is versatility: If needed, one can directly recompute the analysis at a coarser level by aggregating relevance scores according to a partition of the pixel space [133].”
Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, methodology): “To that end, Figure 3 justifies our choice by considering Area Over Prediction Curve (AOPC) [28] for {Ωtn`} on CIFAR-100 [14] tasks, which splits 100 classes into 10 tasks.”
Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks (2016, neutral): “This observation agrees with a recent evaluation of deconv (Samek et al., 2015).”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “For example, for a probabilistic classification model, we can obscure or remove features, measure the drop in probability of the predicted class, and compare against the interpreter’s own prediction of relevance [17, 3].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, methodology): “We can also consider the behavior of these measures as a function of k, inspired by the measurements proposed in Samek et al. (2016) in the context of evaluating saliency maps for image classification.”