Sanity Checks for Saliency Maps

By Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, Been Kim. 2018

2,525 citations recorded by the source corpus.

Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In [263,294] , the authors empirically showed how local explanations in deep networks are strongly dominated by their lower level features.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Such issues with explanations have arisen with saliency maps (Adebayo et al., 2018; Chen et al., 2019), assessment of fairness and variable importance (Rudin et al., 2020; Dimanov et al., 2020), and uncertainty bands for variable importance (Gosiewska and Biecek, 2020; Fisher et al., 2019), as well…”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “This further exposes the fragility of interpretability meth-ods (Adebayo et al., 2018).”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “However, many of these methods have been poorly evaluated so far (Adebayo et al., 2018; Nie et al., 2018), casting doubt on their usefulness in practice.”
A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, background): “Posthoc explanations can be problematic; for instance saliency maps highlight regions of the image, but can be unreliable and misleading, as they tend to highlight edges and do not show what computation is actually done with the highlighted pixels [40, 2, 3].”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, the reliability of existing methods is an active area of research [85] [86] [87] [88] [89] [90] [91] [92] [93].”
Concept whitening for interpretable image recognition (2020, methodology): “n images, regardless of the class. Thus, very similar explanations are given for multiple classes, and often none of them are useful explanations. Saliency methods can be unreliable and fragile (e.g.,Adebayo et al., 2018). Other work provides explanations of how the network’s latent features operate”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, methodology): “Assumption (a) is still subject to very active research, particularly for deep learning methods [31] (see the supplement).”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “Some prior work has evaluated the applicability of saliency maps for explaining the behavior of image classiﬁers (Samek et al., 2017; Adebayo et al., 2018; Kindermans et al., 2019), but there is not a corresponding literature evaluating the applicability of saliency maps for explaining RL agent…”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, background): “, vanilla Backpropagation and Grad-CAM pass important “sanity checks” regarding their sensitivity to data and model parameters (Adebayo et al., 2018).”
Is Attention Interpretable? (2019, background): “Others have explored alternative ways of comparing the behavior of proposed explanation methods (Adebayo et al., 2018).”
Explanations can be manipulated and geometry is to blame (2019, methodology): “To this end, we use three metrics following [23]: the structural similarity index (SSIM), the Pearson correlation coefficient (PCC) and the mean squared error (MSE).”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, methodology): “However, we show that SmoothGrad-Squared (an unpublished variant of Classic SmoothGrad [34]) and Vargrad [2], methods which ensemble a set of estimates produced by basic methods, far outperform both the underlying method and a random guess.”
Counterfactual Visual Explanations (2019, background): “…are increasingly been deployed in real world applications, interpretability of machine learning systems (particularly deep learning models) has become an active area of research (Simonyan et al., 2013; Springen-berg et al., 2015; Ribeiro et al., 2016; Adebayo et al., 2018; Doshi-Velez & Kim, 2017).”
Definitions, methods, and applications in interpretable machine learning (2019, background): “While feature importance scores can provide useful insights, they also have a number of limitations (80, 82).”
Distilling Model Failures as Directions in Latent Space (2022, neutral): “These include saliency maps (Adebayo et al., 2018; Simonyan et al., 2013), integrated gradients (Sundararajan et al., 2017), and LIME (Ribeiro et al., 2016b), and perform feature attribution for particular inputs.”