Investigating the influence of noise and distractors on the interpretation of neural networks

By Pieter-Jan Kindermans, Kristof T. Schütt, K. Müller, Sven Dähne. 2016

131 citations recorded by the source corpus.

Understanding neural networks is becoming increasingly important. Over the last few years different types of visualisation and explanation methods have been proposed. However, none of them explicitly considered the behaviour in the presence of noise and distracting elements. In this work, we will show how noise and distracting dimensions can influence the result of an explanation model. This gives a new theoretical insights to aid selection of the most appropriate explanation model within the deep-Taylor decomposition framework.

Subfields: Artificial Intelligence, Biophysics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Safety, Robustness & Interpretability

Citations in the corpus

Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “…et al., 2017; Robnik-ˇSikonja and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or inﬂuence (Kindermans et al., 2016; Adler et al., 2016; Koh and Liang, 2017), aims to reveal components of high importance in the input to the DNN and their…”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, background): “It can be shown that for ReLU and max-pooling networks, the z -rule reduces to the element-wise multiplication of the input and the saliency map (Shrikumar et al., 2016; Kindermans et al., 2016).”
Learning Important Features Through Propagating Activation Differences (2017, background): “Shrikumar et al. and Kindermans et al. (Shrikumar et al., 2016; Kindermans et al., 2016) showed that absent modifications to deal with numerical stability, the LRP rules for ReLU networks were equivalent within a scaling factor to an elementwise product between the saliency maps of Simonyan et al.…”
A unified view of gradient-based attribution methods for Deep Neural Networks (2017, methodology): “This proposition has been previously proven in the case of Rectiﬁed Linear Unit (ReLU) nonlinearities [17, 4].”
Opportunities and obstacles for deep learning in biology and medicine (2017, methodology): “[336] proposed a strategy called Layerwise Relevance Propagation, which was shown to be equivalent to the element-wise product of the gradient and input [169,337].”
The (Un)reliability of saliency methods (2017, background): “Under specific circumstances, LRP is also equivalent to the gradient times input (Kindermans et al., 2016; Shrikumar et al., 2016).”
Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs (2020, background): “(5) in the main paper) using mathematical induction [11].”
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation (2018, methodology): “Another line of research approximates the model to be interpreted via a locally additive model in order to explain the difference between the model output and some “reference” output in terms of the difference between the input and some “reference” input (Bach et al., 2015; Kindermans et al., 2016; ”
Gradient-Based Attribution Methods (2019, result): “This relation has been previously proven in the literature [11, 28].”
A Diagnostic Study of Explainability Techniques for Text Classification (2020, methodology): “We also select two widely used improvements of the Saliency technique, namely InputX-Gradient (Kindermans et al., 2016), and Guided Backpropagation (Springenberg et al., 2014).”
Full-Gradient Representation for Neural Network Visualization (2019, result): “Similar to our paper, some works [3, 14] also make the observation that including biases within attributions can enable gradient-based attributions to satisfy the completeness property.”
Debugging Tests for Model Explanations (2020, methodology): “For example, for a DNN with all ReLU activations, LRP-Z, a variant of layer-wise relevance propagation that we discuss in the modified back-propagation section is equivalent to Input-Grad (Kindermans et al., 2016).”
Towards better understanding of gradient-based attribution methods for Deep Neural Networks (2017, unknown): “This relation has been previously proven by Shrikumar et al. (2016) and Kindermans et al. (2016).”
Sanity Checks for Saliency Metrics (2019, unknown): “2 It was noted by (Kindermans et al. 2016; Shrikumar, Green-side, and Kundaje 2017) that gradient (cid:12) input was functionally equivalent to a version of the Layerwise Relevance Propagation method, (cid:15) -LRP, (Bach et al. 2015; Kindermans et al. 2016) under certain conditions.”
Fooling Neural Network Interpretations via Adversarial Model Manipulation (2019, methodology): “Note the interpretation methods we used employ formula that are related to some form of gradients; SimpleG is using the gradient of the input, Grad-CAM is a function of the gradient of the representation at certain layer, and LRP turns out to be similar to gradient times inputs [15].”