Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers

By Alexander Binder, Grgoire Montavon, Sebastian Lapuschkin, K. Müller, Wojciech Samek. 2016

569 citations recorded by the source corpus.

Layer-wise relevance propagation is a framework which allows to decompose the prediction of a deep neural network computed over a sample, e.g. an image, down to relevance scores for the single input dimensions of the sample such as subpixels of an image. While this approach can be applied directly to generalized linear mappings, product type non-linearities are not covered. This paper proposes an approach to extend layer-wise relevance propagation to neural networks with local renormalization layers, which is a very common product-type non-linearity in convolutional neural networks. We evaluate the proposed method for local renormalization layers on the CIFAR-10, Imagenet and MIT Places datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Prior works [6, 63, 1, 14] using transformer networks in natural language have re-purposed the attention weights in the later layers as an mechanism to introspect model logic.”
Knowledge Neurons in Pretrained Transformers (2021, methodology): “Besides, a set of attribution methods (Shrikumar et al., 2017; Binder et al., 2016; Zeiler and Fergus, 2014; Springenberg et al., 2015) back-propagate the ﬁnal output to input features.”
Axiomatic Attribution for Deep Networks (2017, methodology): “We now discuss intuition for why DeepLift and LRP break Implementation Invariance; a concrete example is provided in Appendix B.”
Are Sixteen Heads Really Better than One? (2019, methodology): “Their approach involves using LRP (Binder et al., 2016) for determining important heads and looking at specific properties such as attending to adjacent positions, rare words or syntactically related words.”
Methods for interpreting and understanding deep neural networks (2017, background): “Although some propagation rules have been shown to work well in practice for some of them [5, 12, 4], their handling remains to some degree heuristic, and it is still an open question whether the deep Taylor decomposition framework can also be extended to these layers.”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “These layers are treated with an approach based on Taylor expansion [139].”
Understanding intermediate layers using linear classifier probes (2016, methodology): “In Binder et al. (2016) the authors apply the concept of Relevance Propagation to a larger family of models.”
Did the Model Understand the Question? (2018, methodology): “We could use many other methods instead of Integrated Gradients (IG) to attribute a deep network’s prediction to its input features (Baehrens et al., 2010; Simonyan et al., 2013; Shrikumar et al., 2016; Binder et al., 2016; Springenberg et al., 2014).”
Learning Explainable Models Using Attribution Priors (2019, methodology): “There have been many previous attribution methods proposed for deep learning models [18, 3, 33, 37].”
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer (2020, background): “Previous work has explored attributing predictions to the input features with various saliency measures, such as DeepLift (Shrikumar, Greenside, and Kundaje 2017), layerwise relevance propagation (Binder et al. 2016), and Integrated Gradients (IG; Sundararajan, Taly, and Yan 2017).”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, unknown): “The attribution methods are widely studied (Baehrens et al., 2010; Binder et al., 2016; Montavon et al., 2018).”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “LRP and Variants LRP [16, 27, 118], Contrastive LRP [67], Softmax-Gradient LRP [79], RAP [123], Chefer et al.”
How Important Is a Neuron? (2018, background): “The problem of attributing a deep network’s prediction to its input is well-studied (cf. [1, 2, 3, 4, 5, 6, 7]).”
Visualization of Supervised and Self-Supervised Neural Networks via Attribution Guided Factorization (2020, background): “LRP (Binder et al. 2016) can be written in this notation by setting X = x and Θ = θ, where τ = max(0, τ) for a tensor τ .”
Deep learning for situational understanding (2017, methodology): “Recently, the layer-wise relevance propagation algorithm [26], [27] (LRP), has been proposed that uses this property of the individual units to decompose the output of a deep neural network in terms of its input variables and provide transparency into which features of the data were used for the mod”