Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values

By Julius Adebayo, Justin Gilmer, Ian J. Goodfellow, Been Kim. 2018

138 citations recorded by the source corpus.

Explaining the output of a complicated machine learning model like a deep neural network (DNN) is a central challenge in machine learning. Several proposed local explanation methods address this issue by identifying what dimensions of a single input are most responsible for a DNN's output. The goal of this work is to assess the sensitivity of local explanations to DNN parameter values. Somewhat surprisingly, we find that DNNs with randomly-initialized weights produce explanations that are both visually and quantitatively similar to those produced by DNNs with learned weights. Our conjecture is that this phenomenon occurs because these explanations are dominated by the lower level features of a DNN, and that a DNN's architecture provides a strong prior which significantly affects the representations learned at these lower layers. NOTE: This work is now subsumed by our recent manuscript, Sanity Checks for Saliency Maps (to appear NIPS 2018), where we expand on findings and address concerns raised in Sundararajan et. al. (2018).

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “In [263,294] , the authors empirically showed how local explanations in deep networks are strongly dominated by their lower level features.”
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, background): “SmoothGrad [40] and VarGrad [1] visually sharpen explanations by combining multiple explanations of noisy copies of the image.”
Sanity Checks for Saliency Maps (2018, background): “7 VarGrad Similar to SmoothGrad, and as referenced in [37] a variance analog of SmoothGrad can be defined as follows: Evg(x) = V(E(x+ gi)), where noise vectors gi ∼ N (0, σ(2))) are drawn i.”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, background): “Furthermore, some recent work has demonstrated that saliency maps produced by randomized networks are similar to that of the trained network (Adebayo et al., 2018), while simple meaningless data processing steps, such as mean shift, may cause saliency methods to result in significant changes (Kinder”
A Benchmark for Interpretability Methods in Deep Neural Networks (2018, methodology): “However, we show that SmoothGrad-Squared (an unpublished variant of Classic SmoothGrad [34]) and Vargrad [2], methods which ensemble a set of estimates produced by basic methods, far outperform both the underlying method and a random guess.”
Plant disease identification using explainable 3D deep learning on hyperspectral images (2019, background): “The architecture of the convolutional neural network can have a strong prior on the feature importance estimation [49], and this could be more problematic in noisy saliency maps.”
A Note about: Local Explanation Methods for Deep Neural Networks lack Sensitivity to Parameter Values (2018, methodology): “Rather, we attempt to demonstrate that Adebayo et al. (2018) does not seem to show that they are not.”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “[324, 325] compared the saliency maps generated by various methods for visual explainability to either a randomly initialised untrained network or from a copy of the dataset in which the labels were randomly permutated.”
Interpreting Super-Resolution Networks with Local Attribution Maps (2020, background): “For FSRCNN, the range of width is selected to be [16, 128] and the range of depth is [2, 16], with 15 models in total.”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, unknown): “8) Blast Furnace Slag (x[1] = +1.”
Drug discovery with explainable artificial intelligence (2020, methodology): “72 Its popularity arises partially from the fact that this computation can be performed via backpropagation, 73 the main way of computing partial ﬁrst-order derivatives in neural network models.”
Explainable Sentiment Analysis with Applications in Medicine (2018, background): “There are some conflicting views regarding the lackness in sensitivity of local explanation methods [55], [56].”
Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation (2019, background): “Adebayo et al. (2018) argue similarly and show cases when the input term can dominate the interpretation.”
Evaluating Feature Importance Estimates (2018, methodology): “SmoothGradSquared (unpublished variant of Classic SmoothGrad) and Vargrad (Adebayo et al., 2018a) produced large gains in accuracy, while Classic SmoothGrad (Smilkov et al., 2017)), is less accurate or on par with a single estimate but carries a far higher computational burden.”
Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks (2019, background): “Other works [1, 8, 17, 18, 20] build upon and manipulate this gradient to visually sharpen the result.”
Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents (2019, background): “An interesting observation is that such methods seem to generate believable saliency maps even for networks with random weights [1].”
Free-Lunch Saliency via Attention in Atari Agents (2019, methodology): “The current state of the art approaches are SmoothGrad [35] and VarGrad [1], which were studied theoretically in [32] and found to be best in a recent comprehensive evaluation [20].”