Learning how to explain neural networks: PatternNet and PatternAttribution

By Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, K. Müller, Dumitru Erhan, Been Kim, Sven Dähne. 2017

371 citations recorded by the source corpus.

DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Similarly, the authors in [61] analyzed the correctness of current feature relevance explanation approaches designed for Deep Neural Networks, e,g., De-ConvNet, Guided BackProp and LRP, on simple linear neural networks.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “Furthermore, interpretability method such as LRP, deconvolution and guided backpropagation introduced earlier are shown to not work for simple model, such as linear model, bringing into question their reliability [102].”
Sanity Checks for Saliency Maps (2018, methodology): “Aside gradient-based approaches, other methods ‘learn’ an explanation per sample for a model [20, 17, 12, 15, 11, 21].”
Interpreting CNNs via Decision Trees (2018, background): “The concept of the contribution was also called the “importance” in [24, 22] and was termed “attribution” in [16].”
Visual interpretability for deep learning: a survey (2018, methodology): “Studies of [Zintgraf et al. , 2017; Kindermans et al. , 2017] have invented methods to visualize areas in the input image that contribute the most to the decision-making process of the CNN. [Wang et al. , 2017; Goyal et al. , 2016] have tried to interpret the logic for visual question-answering…”
The (Un)reliability of saliency methods (2017, methodology): “, 2015) and PatternNet (Kindermans et al., 2017) aim to isolate input patterns that stimulate neuron activation in higher layers.”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, methodology): “…BagNets which compare against random maskings as well as several popular attribution techniques (Baehrens et al., 2010; Sundararajan et al., 2017; Kindermans et al., 2018; Shrikumar et al., 2017) (we use the implementations of DeepExplain (Ancona et al., 2017)) which compute heatmaps directly in…”
Explanations can be manipulated and geometry is to blame (2019, methodology): “We refer to the original publication for more details [17].”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, methodology): “Closely related to this manuscript are the modiﬁcation-based evaluation measures proposed originally by [29] with subsequent variations [19, 25].”
Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation (2019, background): “…networks (Simonyan et al., 2014; Zeiler & Fergus, 2014; Springenberg et al., 2014; Bach et al., 2015; Ribeiro et al., 2016; Selvaraju et al., 2016; Shrikumar et al., 2017; Sun-dararajan et al., 2017; Montavon et al., 2017; Zintgraf et al., 2017; Lundberg & Lee, 2017; Kindermans et al., 2018).”
A Survey of Deep Learning for Scientific Discovery (2020, background): “Many other techniques look at computing some kind of input mask, several of them using deconvolutional layers, ﬁrst proposed by [263] and built on by [106] and [20].”
Debugging Tests for Model Explanations (2020, methodology): “PatternNet (PNet) and Pattern Attribution (PAttribution) [30] decompose the input into signal and noise components, and back-propagate relevance for the signal component.”
Machine Learning Interpretability: A Survey on Methods and Metrics (2019, methodology): “There are many notorious examples of such DNN explanation methods [7], most of which are used in computer vision [125], including guided backpropagation [126], integrated gradients [127], SmoothGrad saliency maps [128], Grad-CAM [129], and more recently, testing with Concept Activation Vectors (TCAV”
Deep Learning Techniques for Music Generation - A Survey (2017, methodology): “An example of such an approach is using saliency maps with the three following categories101: • gradient sensitivity, to estimate how a small change to the input can affect the classification task (see, for example, [5]); • signal methods, to isolate input patterns that stimulate neuron activation i”
iNNvestigate neural networks! (2018, background): “In particular it contains reference implementations for many methods (PatternNet, PatternAttribution, LRP) and example application for a large number of state-of-the-art applications.”