Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models

By Wojciech Samek, Thomas Wiegand, K. Müller. 2017

1,493 citations recorded by the source corpus.

With the availability of large databases and recent improvements in deep learning methodology, the performance of AI systems is reaching or even exceeding the human level on an increasing number of complex tasks. Impressive examples of this development can be found in domains such as image classification, sentiment analysis, speech understanding or strategic game playing. However, because of their nested non-linear structure, these highly successful machine learning and artificial intelligence models are usually applied in a black box manner, i.e., no information is provided about what exactly makes them arrive at their predictions. Since this lack of transparency can be a major drawback, e.g., in medical applications, the development of methods for visualizing, explaining and interpreting deep learning models has recently attracted increasing attention. This paper summarizes recent developments in this field and makes a plea for more interpretability in artificial intelligence. Furthermore, it presents two approaches to explaining predictions of deep learning models, one method which computes the sensitivity of the prediction with respect to changes in the input and one approach which meaningfully decomposes the decision in terms of the input variables. These methods are evaluated on three classification tasks.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “In a concurrent development, deep learning has revolutionized the ways that information can be extracted from data resources with groundbreaking successes in areas such as computer vision, natural language processing, or voice recognition, among many others [4]–[9].”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “Consequently, new models fusing white- and black-box approaches are necessary [150]–[152].”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “We will not attempt to cover all the related works many of which are already presented in the research and survey works we cite [1] [2] [9] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23].”
Explainable Recommendation: A Survey and New Perspectives (2018, background): “cial to read other related surveys such as explanations in recommender systems from a user study perspective (Tintarev and Masthoff, 2007a), interpretable machine learning (Lipton, 2018; Molnar, 2019), as well as explainable AI in general (Gunning, 2017; Samek et al., 2017).”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, background): “Fortunately, these problems have not escaped the attention of the ML/Deep Learning (DL) community [58, 48, 50, 41, 23, 38, 35, 14].”
Deep Learning: A Critical Appraisal (2018, background): “The relative opacity of “black box” neural networks has been a major focus of discussion in the last few years (Samek, Wiegand, & Müller, 2017; Ribeiro, Singh, & Guestrin, 2016).”
ConvNets and ImageNet Beyond Accuracy: Explanations, Bias Detection, Adversarial Examples and Model Criticism (2017, background): “In order to promote the adoption of such algorithms and to foster their positive technological impact, recent studies have been focusing on understanding a model’s behavior from a human point of view [30, 39, 31, 10].”
Explaining Explanations in AI (2018, background): “…of a system (i.e. as a type of ‘debugging’; (Kulesza et al., 2015)) and help developers and humans working with a system learn from it (Samek et al., 2017), and enhance the trust between individuals subject to a decision and the system itself (Citron and Pasquale, 2014; Hildebrandt…”
Towards Explainable NLP: A Generative Explanation Framework for Text Classification (2018, background): “Some previous works explain the result in two ways: evaluating the sensitivity of output if input changes and analyzing the result from a mathematical perspective by redistributing the prediction function backward (Samek et al., 2018).”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “LRP and Occlusion Sensitivity performed better than Sensitivity Analysis, seconding the findings of [329, 330].”
Explainable Agents and Robots: Results from a Systematic Literature Review (2019, background): “Thus, the majority of the recent studies and surveys focus on providing an overview on interpretability and explainability of data-driven algorithms [5, 24, 35, 72, 86].”
Blockchain for AI: Review and Open Research Challenges (2019, background): “Table 1 shows few emerging trends in AI applications such as explainable AI [9], [30], digital twins [31], automated machine learning [32], hybrid learning models [33], and lean and augmented data learning [34], and the perceived benefits”
Evaluating adversarial attacks against multiple fact verification systems (2019, background): “It has been observed in related NLP tasks that as models become more complex, it is difficult to fully understand and characterize their behaviour (Samek et al., 2017).”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2021, methodology): “They showed that the heatmaps produced by SA are much more noisy compared to the heatmaps generated using LRP [65].”
Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks (2018, background): “computation, [42, 44] include further stabilizers as:”
Improved Code Summarization via a Graph Neural Network (2020, methodology): “To help us answer RQ 3 we use concepts similar to those outlined in Samek et al. [47] for model visualizations.”
An Interpretable Generative Model for Handwritten Digits Synthesis (2019, background): “Explanable machine learning (XML) [3, 4] and explainable Artificial Intelligence (XAI) [5, 6] have attracted a lot of attention recently.”
Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challenges (2019, background): “XAI is a branch of AI (and its descendant ML and DL methods) that break the black-box nature of the inner layers of these model and make it transparent, and thus more reliable (or at least understandable) [190], [191].”
Evolutionary Fuzzy Systems for Explainable Artificial Intelligence: Why, When, What for, and Where to? (2019, background): “In recent years has emerged the broader concept explainability in what has been called eXplanaible Artificial Intelligence (XAI) that encompasses ML/AI systems for opening black box models, for improving the understanding of what the models have learned and/or for explaining individual predictions [”