Not Just a Black Box: Learning Important Features Through Propagating Activation Differences

By Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, Anshul B Kundaje. 2016

887 citations recorded by the source corpus.

Note: This paper describes an older version of DeepLIFT. See this https URL for the newer version. Original abstract follows: The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Learning Important FeaTures), an efficient and effective method for computing importance scores in a neural network. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. We apply DeepLIFT to models trained on natural images and genomic data, and show significant advantages over gradient-based methods.

Subfields: Artificial Intelligence, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

Axiomatic Attribution for Deep Networks (2017, methodology): “The need for a baseline has also been pointed out by prior work on attribution (Shrikumar et al., 2016; Binder et al., 2016).”
Understanding Black-box Predictions via Influence Functions (2017, background): “By understanding why a model does what it does, we can hope to improve the model (Amershi et al., 2015), discover new science (Shrikumar et al., 2016), and provide end-users with explanations of actions that impact them (Goodman & Flaxman, 2016).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “In the same direction, the authors in [110] proposed DeepLIFT, an approach for computing importance scores in a multi-layer neural network.”
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, neutral): “, 2010; Rieger & Hansen, 2019), decomposition-based (Murdoch & Szlam, 2017; Shrikumar et al., 2016; Bach et al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et al.”
Sanity Checks for Saliency Maps (2018, background): “SmoothGrad (SG) [16] seeks to alleviate noise and visual diffusion [14, 13] for saliency maps by averaging over explanations of noisy copies of an input.”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, background): “SmoothGrad (SG) (Smilkov et al., 2017) seeks to alleviate noise and visual diffusion (Sundararajan et al., 2017; Shrikumar et al., 2016) for saliency maps by averaging over explanations of noisy copies of an input.”
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, neutral): “Not surprisingly, the development of techniques for “open-ing” black box models has recently received a lot of attention in the community [6, 35, 39, 5, 33, 25, 23, 30, 40, 11, 27].”
A Unified Approach to Interpreting Model Predictions (2017, methodology): “DeepLIFT: DeepLIFT was recently proposed as a recursive prediction explanation method for deep learning [7, 6].”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, background): “It can be shown that for ReLU and max-pooling networks, the z -rule reduces to the element-wise multiplication of the input and the saliency map (Shrikumar et al., 2016; Kindermans et al., 2016).”
Learning Important Features Through Propagating Activation Differences (2017, background): “Shrikumar et al. and Kindermans et al. (Shrikumar et al., 2016; Kindermans et al., 2016) showed that absent modiﬁcations to deal with numerical stability, the original LRP rules were equivalent within a scaling factor to an elementwise product between the saliency maps of Simonyan et al. and the…”
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, methodology): “…from the decision boundary, they can be uniformatively small and do not always capture the idea of salience (see discussion and alternatives proposed in [Shrikumar et al., 2016; Bach et al., 2015; Montavon et al., 2017; Sundararajan et al., 2016; Shrikumar et al., 2017; Fong and Vedaldi, 2017]).”
A unified view of gradient-based attribution methods for Deep Neural Networks (2017, methodology): “Gradient * Input [18] was at first proposed as a technique to improve the sharpness of attribution maps.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, neutral): “, 2017b), Gradient Input (Shrikumar et al., 2016), SmoothGrad etc.”
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis (2017, methodology): “Other methods include Shrikumar et al. (2016), who compare the activation of a unit when a speciﬁc input is fed forward through the net to a reference activation for that unit.”
The (Un)reliability of saliency methods (2017, background): “Under specific circumstances, LRP is also equivalent to the gradient times input (Kindermans et al., 2016; Shrikumar et al., 2016).”
An unexpected unity among methods for interpreting model predictions (2016, methodology): “DeepLIFT was recently proposed as a recursive prediction explanation method for deep learning [5].”
Interpreting Blackbox Models via Model Extraction (2017, methodology): “Similarly, [39] proposes a method for computing inﬂuence scores for features in deep neural nets.”
Explaining Explanations in AI (2018, methodology): “…should we only pay attention to contrasting solutions closest to the original solution as suggested by DeepLift (illustrated by Figure 1 centre; (Shrikumar et al., 2017, 2016)); uniformly weight over all possible values, suggested by SHAPE (Lundberg and Lee, 2017) (Figure 1 right); or pay as…”
Did the Model Understand the Question? (2018, methodology): “We could use many other methods instead of Integrated Gradients (IG) to attribute a deep network’s prediction to its input features (Baehrens et al., 2010; Simonyan et al., 2013; Shrikumar et al., 2016; Binder et al., 2016; Springenberg et al., 2014).”