Axiomatic Attribution for Deep Networks
By Mukund Sundararajan, Ankur Taly, Qiqi Yan. 2017
8,293 citations recorded by the source corpus.
We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works. We identify two fundamental axioms— Sensitivity and Implementation Invariance that attribution methods ought to satisfy. We show that they are not satisfied by most known attribution methods, which we consider to be a fundamental weakness of those methods. We use the axioms to guide the design of a new attribution method called Integrated Gradients. Our method requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator. We apply this method to a couple of image models, a couple of text models and a chemistry model, demonstrating its ability to debug networks, to extract rules from a network, and to enable users to engage with models better.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
- Noah A. Smith
- Caiming Xiong
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Yonatan Belinkov
- Noah A. Smith
- Yejin Choi
- Christopher D. Manning
Citations in the corpus
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Li et al. (2016) examined the impact of erasing portions of a network’s representations on the output, Sundararajan et al. (2017) used a gradient based method to attribute predictions to inputs, and Murdoch et al. (2018) decomposed LSTMs to interpret classification predictions.”
- SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “Post-hoc Interpretation Methods: Predominant based methods for post-hoc interpretability in NLP use gradient based methods (Simonyan et al., 2014; Sundararajan et al., 2017; Smilkov et al., 2017).”
- Refining Language Models with Compositional Explanations (2021, methodology): “With model-agnostic post-hoc explanation algorithms that explain predictions of a trained model without interfering with its learned weights [42, 19], our interactive machine learning method enables inspection and feedback to models as complex as BERT.”
- WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “This has led to a substantial body of research that endeavors to make neural networks more “interpretable”, e.g. by attributing its prediction to a given part of its input (Baehrens et al., 2010; Sundararajan et al., 2017; ar X iv :2 00 4.”
- Locating and Editing Factual Associations in GPT (2022, methodology): “In particular, we find that Causal Tracing is more informative than gradient-based salience methods such as integrated gradients (Sundararajan et al., 2017) (Figure 16) and is robust under different noise configurations.”
- Knowledge Neurons in Pretrained Transformers (2021, methodology): “Inspired by Hao et al. (2021), we propose a knowledge attribution method based on integrated gradients (Sundararajan et al., 2017).”
- Analysis Methods in Neural Language Processing: A Survey (2018, background): “The important or salient features can then be visualized in selected examples (Li et al., 2016a; Aubakirova and Bansal, 2016; Sundararajan et al., 2017; Arras et al., 2017a,b; Ding et al., 2017; Murdoch et al., 2018; Mudrakarta et al., 2018; Montavon et al., 2018; Godin et al., 2018).”
- What Does BERT Look at? An Analysis of BERT’s Attention (2019, methodology): “To further investigate this hypothesis, we apply gradient-based measures of feature importance (Sundararajan et al., 2017).”
- Dynabench: Rethinking Benchmarking in NLP (2021, methodology): “1 The platform not only displays prediction probabilities, but through an “inspect model” functionality, allows the user to examine the token-level layer integrated gradients (Sundararajan et al., 2017), obtained via the Captum interpretability library.”
- Attention is not Explanation (2019, methodology): “These include approaches that measure feature importance based on gradient information (Ross et al., 2017; Sundararajan et al., 2017) (aligned with the gradient-based measures that we have used here), and methods based on representation erasure (Li et al., 2016), in which dimensions are removed and…”
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Following these axioms, the authors of [262] created integrated gradients , a new feature relevance method proven to meet the aforementioned axioms.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “Examples are LRP [40], DeepLIFT [41], CAM [42], Grad-CAM [43], Integrated gradients [44], and SmoothGrad [45].”
- Analyzing Federated Learning through an Adversarial Lens (2018, background): “, Integrated Gradients ((Sundararajan et al., 2017)), or generating sensitivity and saliency maps (SmoothGrad ((Smilkov et al.”
- Model Fusion via Optimal Transport (2019, background): “Although we note that it is possible to additionally use other measures of neuron importance [20, 21], but we leave it for a future work.”
- Explaining Models by Propagating Shapley Values of Local Components (2019, background): “…numerous works: Occlusion (Zeiler and Fergus 2014), Saliency Maps (Simonyan, Vedaldi, and Zisserman 2013), Layer-Wise Relevance Propagation (Bach et al. 2015), DeepLIFT, Integrated Gradients (IG) (Sundararajan, Taly, and Yan 2017), and Generalized Integrated Gradients (GIG) (Merrill et al. 2019).”
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “They can be categorized as gradient-based (Springenberg et al., 2014; Sundararajan et al., 2017; Selvaraju et al., 2016; Baehrens et al., 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong &…”
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “Integrated Gradients [68], [93] 7 X Invariance Input invariance [70] 7 X”
- Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, background): “Integrated Gradients [43] additionally accumulates gradients along a path from a base image to the input image.”
- EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “The most famous method is LIME [20], but other method exist [4, 21, 23, 24].”