A unified view of gradient-based attribution methods for Deep Neural Networks

By Marco Ancona, Enea Ceolini, Cengiz Oztireli, Markus H. Gross. 2017

149 citations recorded by the source corpus.

Understanding the flow of information in Deep Neural Networks is a challenging problem that has gain increasing attention over the last few years. While several methods have been proposed to explain network predictions, only few attempts to analyze them from a theoretical perspective have been made in the past. In this work we analyze various state-of-the-art attribution methods and prove unexplored connections between them. We also show how some methods can be reformulated and more conveniently implemented. Finally, we perform an empirical evaluation with six attribution methods on a variety of tasks and architectures and discuss their strengths and limitations.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, neutral): “For example, [46] conducts an”
Mitigating Sybils in Federated Learning Poisoning (2018, background): “Recent work on feature influence in deep neural networks [1], [15] may better capture the intent of sybil-based poisoning attacks in deep learning and we leave this analysis as future work.”
Sanity Checks for Saliency Maps (2018, methodology): “[24] showed that for ReLU networks (with zero baseline and no biases) the -LRP and DeepLift (Rescale) explanation methods are equivalent to the input gradient.”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “Attribution, a term introduced by [2], also referred to as relevance [5, 7, 60, 47], contribution [51], class saliency [52] or influence [27, 1, 28], aims to reveal components of high importance in the input to the DNN and their effect as the input is propagated through the network.”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “…obtaining local-explanations (Ribeiro et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Pieter-Jan Kindermans, 2018), recent work by Ancona et al. (2017) and Lundberg & Lee (2017) both show that there are equivalences among several of the previously proposed methods.”
On the Binding Problem in Artificial Neural Networks (2020, background): “However, it has repeatedly been shown that such features are best understood as “a texture detector highly correlated with an object” (Olah et al., 2020; Sundararajan et al., 2017; Ancona et al., 2017; Brendel and Bethge, 2019; Geirhos et al., 2019).”
The Limitations of Federated Learning in Sybil Settings (2020, background): “Recent work on feature influence in deep neural networks [1, 16, 32] may better capture the intent of sybil-based poisoning attacks in deep learning and we leave this analysis as future work.”
On the Privacy Risks of Model Explanations (2019, methodology): “ to additional information, such as active access to model queries (e.g. [1, 10, 24]), a prior over the data distribution [5], or knowledge of the model class (e.g. that the model is a neural network [3, 28, 34], or that we know the source code [9]). The output of an explanation function ϕ(X,c,xﬁ0,·”
On the (In)fidelity and Sensitivity for Explanations. (2019, background): “We note that our definition of local and global explanations follows the description of [3], distinct from that in [30].”
Finding Influential Training Samples for Gradient Boosted Decision Trees (2018, methodology): “Some of the applications and benefits of employing these methods outlined in previous work (Ancona et al., 2017) include (1) “debugging” the model to expose ways of model failures not discoverable via conventional test set performance measuring (e.”
Interpretable and Differentially Private Predictions (2019, methodology): “One prominent aspect of these approaches is the use of gradient-based attributions (Selvaraju et al. 2016; Ribeiro, Singh, and Guestrin 2016; Smilkov et al. 2017; Sundararajan, Taly, and Yan 2017; Montavon et al. 2015; Bach et al. 2015; Ancona et al. 2017).”
Explainable AI for medical imaging: deep-learning CNN ensemble for classification of estrogen receptor status from breast MRI (2020, methodology): “…by using methods applicable to any deep-learning convolutional neural network (DCNN) models, and accounting for cross-correlation between input features and diverging from other perturbation-based methods (i.e. occlusion) that only consider the effect of individual features on the prediction[5].”
Privacy Risks of Explaining Machine Learning Models (2019, methodology): “ to additional information, such as active access to model queries (e.g. [1, 10, 24]), a prior over the data distribution [5], or knowledge of the model class (e.g. that the model is a neural network [3, 28, 34], or that we know the source code [9]). The output of an explanation function ϕ(X,c,xﬁ0,·”
Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning (2018, methodology): “A comparison of some of these methods can be found in Ancona [46].”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, methodology): “• DeepExplain package is maintained by Marco Ancona, author of [60].”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “[11] argue that user studies imply a strong bias towards simpler explanations that are closer to the user’s expectations, “at the cost of penalizing those methods that might more closely reflect the network behavior” [11].”
How Sensitive are Sensitivity-Based Explanations? (2019, methodology): “As shown in [4], -LRP [5], Deep LIFT [20], and Integrated Gradients [25] can also be seen as a variant of gradient explanations.”
Sanity Checks for Saliency Metrics (2019, unknown): “Several authors have explored the theoretical relationships between different methods, finding many commonalities between different approaches (Lundberg and Lee 2017; Ancona et al. 2017; 2018).”