Gradients of Counterfactuals

By Mukund Sundararajan, Ankur Taly, Qiqi Yan. 2016

115 citations recorded by the source corpus.

Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score. Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning Important Features Through Propagating Activation Differences (2017, background): “Instead of computing the gradients at only the current value of the input, one can integrate the gradients as the inputs are scaled up from some starting value (eg: all zeros) to their current value (Sundararajan et al., 2016).”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, methodology): “…from the decision boundary, they can be uniformatively small and do not always capture the idea of salience (see discussion and alternatives proposed in [Shrikumar et al., 2016; Bach et al., 2015; Montavon et al., 2017; Sundararajan et al., 2016; Shrikumar et al., 2017; Fong and Vedaldi, 2017]).”
Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients (2017, methodology): “In those cases, smoothing techniques have been developed (Smilkov et al. 2017; Sundararajan, Taly, and Yan 2016) to generate more interpretable explanations, but we have already argued that these techniques may obscure information about the model’s sensitivity to background features.”
LEMNA: Explaining Deep Learning based Security Applications (2018, background): “Recent works [53, 59] attempted to address this problem through approximation.”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “Integrated Gradients (IG) uses interior gradients of deep networks to capture the importance of the different input values (Sundararajan et al., 2016).”
Obtaining genetics insights from deep learning via explainable artificial intelligence (2022, methodology): “Reference-based gradient methods, which include the widely used integrated gradients 23,58 , are designed to address this problem.”
PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks (2020, methodology): “The gradient-based approach has been one of the most popular explanation methods for conventional neural networks [18, 19, 40, 20, 21, 15].”
Explaining Visual Models by Causal Attribution (2019, methodology): “[17] used it to refer to interpolations between an image and a baseline (a blank picture), as a way to estimate the gradient of the classifier in non-saturated regions of the input.”