On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
By Sebastian Bach, Alexander Binder, Grgoire Montavon, Frederick Klauschen, K. Müller, Wojciech Samek. 2015
4,977 citations recorded by the source corpus.
Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. Although machine learning methods are solving very successfully a plethora of tasks, they have in most cases the disadvantage of acting as a black box, not providing any information about what made them arrive at a particular decision. This work proposes a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers. We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks. These pixel contributions can be visualized as heatmaps and are provided to a human expert who can intuitively not only verify the validity of the classification decision, but also focus further analysis on regions of potential interest. We evaluate our method for classifiers trained on PASCAL VOC 2009 images, synthetic image data containing geometric shapes, the MNIST handwritten digits data set and for the pre-trained ImageNet model available as part of the Caffe open source package.
Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Yonatan Belinkov
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
In Biophysics
- Wojciech Samek
- K. Müller
- Grgoire Montavon
- Been Kim
- Sebastian Lapuschkin
- Lars Kai Hansen
- Longo Luca
- Lior Wolf
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Andrea Vedaldi
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
In Safety, Robustness & Interpretability
- Wojciech Samek
- K. Müller
- Longo Luca
- Giulia Vilone
- Been Kim
- Grgoire Montavon
- Su-In Lee
- Scott M. Lundberg
Citations in the corpus
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “Layer-wise relevance propagation (LRP) was originally designed to compute the contributions of single pixels to predictions of image classi-fiers (Bach et al., 2015).”
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “The authors in [72] proposed visualizing the contribution to the prediction of each single pixel of the input image in the form of a heatmap.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “Examples are LRP [40], DeepLIFT [41], CAM [42], Grad-CAM [43], Integrated gradients [44], and SmoothGrad [45].”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Despite intense interest and collective group efforts, we are only seeing limited progress in definitions [8], frameworks [9], visualization [10], and limited experimentation [11].”
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, neutral): “…2017; Selvaraju et al., 2016; Baehrens et al., 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al., 2016; Zintgraf et al., 2017), with many similarities…”
- Generative Counterfactual Introspection for Explainable Deep Learning (2019, methodology): “Many CNN interpretation methods [4]–[6], [12], [13], such as GradCAM [12], utilize backpropagation to conduct sensitivity analysis by attributing the prediction to the input domain (e.”
- Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “Other BBMs such as Layer-wise Relevance Propagation [4], DeepLift [37] or Excitation Backprop [49] utilize top-down relevancy propagation rules.”
- EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “The most famous method is LIME [20], but other method exist [4, 21, 23, 24].”
- Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “(Amos, Xu, and Kolter, 2017; Bach et al., 2015; Baehrens, et al., 2010; Clos, Wiratung, and Massie, 2017; Donahue, et al., 2015; Doshi-Velez and Kim, 2017; Elenberg, Dimakis, Feldman, and Karbasi, 2017; Erhan, Bengio, Courville, and Vincent, 2009; Féraud and Clérot, 2002; Fukui et al., 2016; Gan,…”
- Learning to Explain With Complemental Examples (2018, background): “The primary stream is visualizing where the classifier weighs for its prediction by assigning an importance to each element in the input space, by propagating the prediction to the input space [24, 3, 29, 23, 30, 8, 31], or by learning the instance-wise importance of elements [4, 5, 15] with an auxi”
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “Another technique to compute relevances was proposed in [131] as the Layer-wise Relevance Propagation algorithm.”
- Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “Attribution, a term introduced by Ancona et al. (2018), also referred to as relevance (Bach et al., 2015; Binder et al., 2016; Zintgraf et al., 2017; Robnik-ˇSikonja and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or influence (Kindermans et al.,…”
- Interpreting Deep Classifier by Visual Distillation of Dark Knowledge (2018, methodology): “An alternative is attribution methods that visualize how different parts of the input contribute to the final output, such by generating sensitivity maps over the input (Bach et al., 2015; Baehrens et al., 2010; Selvaraju et al., 2016; Smilkov et al., 2017).”
- A Survey of Methods for Explaining Black Box Models (2018, methodology): “to overcome a functional approach, where the explanation results form the local analysis of the prediction function [109], and where the explanation is obtained by running a backward pass in that graph [7, 137].”
- What do we need to build explainable AI systems for the medical domain? (2017, background): “Bach et al. (2015) presented a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers which allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over bag of words features and for…”
- Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, methodology): “In the following we describe the LRP redistribution process for feed-forward neural networks, redistribution procedures have also been proposed for other popular models [5, 4, 20].”
- Methods for interpreting and understanding deep neural networks (2017, methodology): “Techniques for interpreting and understanding what the model has learned have therefore become a key ingredient of a robust validation procedure [68,22,5].”