Methods for interpreting and understanding deep neural networks

By Grgoire Montavon, Wojciech Samek, K. Müller. 2017

2,553 citations recorded by the source corpus.

Abstract This paper provides an entry point to the problem of interpreting a deep neural network model and explaining its predictions. It is based on a tutorial given at ICASSP 2017. As a tutorial paper, the set of methods covered here is not exhaustive, but sufficiently representative to discuss a number of questions in interpretability, technical challenges, and possible applications. The second part of the tutorial focuses on the recently proposed layer-wise relevance propagation (LRP) technique, for which we provide theory, recommendations, and tricks, to make most efficient use of it on real data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Other than the systems and engineering challenges (Jouppi et al., 2017; Huang et al., 2018; Shazeer et al., 2018), training deep and high capacity neural networks poses signiﬁcant trainabil-ity challenges (Montavon et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “The important or salient features can then be visualized in selected examples (Li et al., 2016a; Aubakirova and Bansal, 2016; Sundararajan et al., 2017; Arras et al., 2017a,b; Ding et al., 2017; Murdoch et al., 2018; Mudrakarta et al., 2018; Montavon et al., 2018; Godin et al., 2018).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “• Understandability (or equivalently, intelligibility ) denotes the characteristic of a model to make a human understand its function – how the model works – without any need for explaining its internal structure or the algorithmic means by which the model processes data internally [18] .”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “A full derivation is provided in Appendix D. Apparently, the search gradient is equivalent to a policy gradient for distribution at this edge whose credit is assigned as From a decision-wise perspective, this reward could be interpreted as contribution analysis of L with Taylor Decomposition (Montav”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “We will not attempt to cover all the related works many of which are already presented in the research and survey works we cite [1] [2] [9] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23].”
Fairwashing: the risk of rationalization (2019, methodology): “Existing methods to achieve explainability include transparent-box design and black-box explanation (also called post-hoc explanation) (Lipton, 2018; Lepri et al., 2017; Montavon et al., 2018; Guidotti et al., 2018).”
Sanity Checks for Saliency Maps (2018, methodology): “[30] proposed an input perturbation procedure for assessing the quality of saliency methods.”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “Fortunately, these problems have not escaped the attention of the ML/Deep Learning (DL) community [58, 48, 50, 41, 23, 38, 35, 14].”
What do we need to build explainable AI systems for the medical domain? (2017, methodology): “The relation between the expert and the resulting prototype is given qualitatively in Figure 1 by Montavon et al. (2017): Here we see four diﬀerent cases: In case a the expert is absent, i.e. the optimization problem reduces to the maximization of the class probability function p ( ω c | x ).”
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (2017, methodology): “Not surprisingly, the development of techniques for “opening” black box models has recently received a lot of attention in the community [6, 35, 39, 5, 33, 25, 23, 30, 40, 11, 27].”
Explaining Recurrent Neural Network Predictions in Sentiment Analysis (2017, methodology): “We refer the reader to (Montavon et al., 2017b) for an overview on methods for understanding and interpreting deep neural network predictions.”
A unified view of gradient-based attribution methods for Deep Neural Networks (2017, methodology): “For a discussion about advantages and drawbacks of gradient-based methods we refer the reader to [17, 13, 23].”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “sis. For decomposing a linear gone can use the Taylor decomposition (a linear approximation) of gwith respect to the h[3, 75]. A non-linear gcan be decomposed by layerwise relevance propagation (LRP) [3,76] or integrated gradients (IG) [112]. Reward Redistribution based on Return Decomposition. The ”
An overview of deep learning in medical imaging focusing on MRI (2018, background): “Considerable effort goes into developing new ways to deal with this problem, including DARPA launching a whole program “Explainable AI ”47 dedicated to this issue, and lots of research going into enhancing interpretability [345,346], and finding new ways to measure sensitivity and visualize features”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, neutral): “In contrast to post-hoc explanations, which approximate a trained model or its output [37, 31], a ProtoTree is inherently interpretable since it directly incorporates interpretability in the structure of the predictive model [37].”
Computing Optimal Decision Sets with SAT (2020, background): “Hence the rising interest in explainable artificial intelligence (XAI) [4, 14, 16, 18, 22, 23, 26, 29, 30, 36–38, 42, 43, 48, 49, 52], including research programs [3, 24] and legislation [17, 21].”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “[2] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.”
Learning Fair Rule Lists (2019, background): “Two main approaches have emerged in the literature to facilitate the understanding of machine learning models: black-box explanation and transparentbox design (Lipton, 2018; Lepri et al., 2017; Montavon et al., 2018; Guidotti et al., 2018).”