Learning Important Features Through Propagating Activation Differences
By Avanti Shrikumar, Peyton Greenside, Anshul B Kundaje. 2017
4,716 citations recorded by the source corpus.
The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, code: http://goo.gl/RM8jvH.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Jianfeng Gao
- Noah A. Smith
- Luke S. Zettlemoyer
- Graham Neubig
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Jitendra Malik
- Andrea Vedaldi
In Safety, Robustness & Interpretability
- Wojciech Samek
- K. Müller
- Been Kim
- Longo Luca
- Giulia Vilone
- Grgoire Montavon
- Su-In Lee
- Lars Kai Hansen
Citations in the corpus
- SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “Approaches that assign relative importance scores to input features through activation differences (Shrikumar et al., 2017; Montavon et al., 2017) are widely adopted for interpretability in computer vision applications.”
- Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Unlike prior work in computer vision (Hooker et al., 2019), we find raw gradients to perform comparably to the input*gradient variant (Shrikumar et al., 2017).”
- Knowledge Neurons in Pretrained Transformers (2021, methodology): “Besides, a set of attribution methods (Shrikumar et al., 2017; Binder et al., 2016; Zeiler and Fergus, 2014; Springenberg et al., 2015) back-propagate the final output to input features.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “Examples are LRP [40], DeepLIFT [41], CAM [42], Grad-CAM [43], Integrated gradients [44], and SmoothGrad [45].”
- Efficient Formal Safety Analysis of Neural Networks (2018, methodology): “Besides formal analysis of safety properties, we believe that our method for efficiently estimating tight and rigorous output ranges of a network for a given input range will also be useful for guiding the training process of robust networks [32, 26] and improving explainability of the decisions mad”
- Explaining Models by Propagating Shapley Values of Local Components (2019, methodology): “Although similar methodologies have been used heuristically (Shrikumar, Greenside, and Kundaje 2017; Erion et al. 2019), we provide a theoretical justification in Theorem 1 in the context of SHAP values.”
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “Other decomposition methods further developed include Deep Taylor Decomposition [44], DeepLIFT and gradient*input [45], Prediction Difference Analysis [46].”
- Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks (2019, methodology): “Other BBMs such as Layer-wise Relevance Propagation [4], DeepLift [37] or Excitation Backprop [49] utilize top-down relevancy propagation rules.”
- EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction (2019, methodology): “The most famous method is LIME [20], but other method exist [4, 21, 23, 24].”
- The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning (2019, methodology): “Furthermore, Kenny and Keane [2019] revisit the techniques tested in this early work and compare them to more recent methods (e.g., DeepLIFT [Shrikumar et al., 2017] and LIME [Riberio et al., 2016]) to see whether they work well in a wide variety of ANN-CBR twins.”
- On the Art and Science of Machine Learning Explanations (2018, methodology): “Popular techniques have names like DeepLIFT and integrated gradients [26], [27], [2].”
- Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “…also referred to as relevance (Bach et al., 2015; Binder et al., 2016; Zintgraf et al., 2017; Robnik-ˇSikonja and Kononenko, 2008), contribution (Shrikumar et al., 2017), class saliency (Simonyan et al., 2013) or influence (Kindermans et al., 2016; Adler et al., 2016; Koh and Liang, 2017), aims…”
- A Survey of Methods for Explaining Black Box Models (2018, background): “Another work following this line of research is Reference [107], which focuses on the activation differences.”
- SmoothGrad: removing noise by adding noise (2017, methodology): “, 2015), DeepLift (Shrikumar et al., 2017), and more recently Integrated Gradients (Sundararajan et al.”
- A Unified Approach to Interpreting Model Predictions (2017, methodology): “These insights also in turn improve DeepLIFT, and a new version includes updates to better match Shapley values [6].”
- Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations (2017, background): “…from the decision boundary, they can be uniformatively small and do not always capture the idea of salience (see discussion and alternatives proposed in [Shrikumar et al., 2016; Bach et al., 2015; Montavon et al., 2017; Sundararajan et al., 2016; Shrikumar et al., 2017; Fong and Vedaldi, 2017]).”
- A unified view of gradient-based attribution methods for Deep Neural Networks (2017, methodology): “For a discussion about advantages and drawbacks of gradient-based methods we refer the reader to [17, 13, 23].”