The (Un)reliability of saliency methods

By Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, Been Kim. 2017

806 citations recorded by the source corpus.

Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. In order to guarantee reliability, we posit that methods should fulfill input invariance, the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy input invariance result in misleading attribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “ical to evaluate whether the gradient-attribution scores truly capture token importance, since these methods can be unreliable for certain datasets or architectures (Kindermans et al., 2019).”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “However, the reliability of some of these methods has been questioned (Kindermans et al., 2019; Hooker et al., 2018; Jain & Wallace, 2019; Serrano & Smith, 2019; Pruthi et al., 2019), limiting their practical util-”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “Methods of interpretability that observe the stimulation of neuron layers like this are also called signal methods [48] (although this paper specifically refers to the observation of neuron layers at higher layers).”
Sanity Checks for Saliency Maps (2018, background): “[26] both present attacks against saliency methods; showing that it is possible to manipulate derived explanations in unintended ways.”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Governmental and non-governmental organizations may put forward legitimate information requests regarding the operations and consequences of DNNs. Stakeholders are often interested in the ethical and legal concerns raised in any phase of the process.”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, background): “…work has demonstrated that saliency maps produced by randomized networks are similar to that of the trained network (Adebayo et al., 2018), while simple meaningless data processing steps, such as mean shift, may cause saliency methods to result in signiﬁcant changes (Kindermans et al., 2017).”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, neutral): “In contrast to local explanations, which explain a single prediction and can be unstable and contradicting [3, 27], global explanations enable simulatability [35].”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, the reliability of existing methods is an active area of research [85] [86] [87] [88] [89] [90] [91] [92].”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “…work has evaluated the applicability of saliency maps for explaining the behavior of image classiﬁers (Samek et al., 2017; Adebayo et al., 2018; Kindermans et al., 2019), but there is not a corresponding literature evaluating the applicability of saliency maps for explaining RL agent behavior.”
Interpretability Beyond Classification Output: Semantic Bottleneck Networks (2019, neutral): “The first being post-hoc interpretation, for which we take an already trained and well performing model and dissect its decisions a-posteriori to identify important input features via attribution [2, 21, 13, 33, 22, 24] or attempt to assign meaning to single or groups of features in order to underst”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “For high-dimensional inputs, raw features (such as individual pixels in images) often lead to noisy explanations that are sensitive to imperceptible artifacts in the data, tend to be hard to analyze coherently and not robust to simple transformations such as constant shifts [9].”
Pathologies of Neural Models Make Interpretations Difficult (2018, background): “Our work is closely related to recent explorations into failures of interpretation methods (Kindermans et al., 2017; Ghorbani et al., 2017).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “, 2019; Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019), saliency methods (Alvarez-Melis and Jaakkola, 2018; Kindermans et al., 2019), or others (Ghorbani et al.”
Seeing What a GAN Cannot Generate (2019, background): “Most prior work on network visualization concerns discriminative classifiers [1, 3, 23, 26, 37, 38, 48, 50].”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, background): “For example, in [18] a unit test for interpretability methods is proposed which detects whether the explanation can be manipulated by factors that are not affecting the decision making process.”
On the Robustness of Interpretability Methods (2018, background): “For example, Kindermans et al. (2017) showed that most saliency methods are not invariant under simple transformations of the input, and are very sensitive to the choice of reference point.”
iNNvestigate neural networks! (2018, background): “Theoretically it is not clear which method solves the stated problems best, therefore an empirical comparison is required (Samek et al., 2017; Kindermans et al., 2017).”
Semantic bottleneck for computer vision tasks (2018, background): “put dimensions involved in the explanation representation. [34] and [35] described geometric metrics to assess the quality of the visual explanation with respect to landmarks or objects in the image. [36] questioned the stability of saliency based visual explanations by showing that a simple constan”