Explaining Explanations: An Overview of Interpretability of Machine Learning

By Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael A. Specter, Lalana Kagal. International Conference on Data Science and Advanced Analytics, 2018

2,256 citations recorded by the source corpus.

There has recently been a surge of work in explanatory artificial intelligence (XAI). This research area tackles the important problem that complex machines and algorithms often cannot provide insights into their behavior and thought processes. XAI allows users and parts of the internal system to be more transparent, providing explanations of their decisions in some level of detail. These explanations are important to ensure algorithmic fairness, identify potential bias/problems in the training data, and to ensure that the algorithms perform as expected. However, explanations produced by these systems is neither standardized nor systematically assessed. In an effort to create best practices and identify open challenges, we describe foundational concepts of explainability and show how they can be used to classify existing literature. We discuss why current approaches to explanatory methods especially for deep neural networks are insufficient. Finally, based on our survey, we conclude with suggested future research directions for explanatory artificial intelligence.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

View Paper: https://arxiv.org/abs/1806.00069

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “[13] ; and (b) its connection to the taxonomy in Fig.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “d in the research papers and survey we cite [1], [2], [15]–[30]. We extend the so-called integrated interpretability [16] by including considerations for subject-content-dependent models. Compared to [17], we also overview the mathematical formulation of common or popular methods, revealing the grea”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…2012, 2011, 2013, 2015; Nataksu, 2004; Ribiero, et al., 2016a,b; Sheh and Monteath, 2008; Treu, 1998; Wickens, 1994), and the challenges for the evaluation of explanation systems (Doshi-Velez and Kim, 2017ab; Gilpin et al., 2018; Feldmann, 2018; Miller, 2017; Miller, Howe, and Sonenberg, 2017).”
Weakly Supervised Disentanglement with Guarantees (2019, background): “Learning such representations that align with the underlying factors of variation may be critical to the development of machine learning models that are explainable or human-controllable (Gilpin et al., 2018; Lee et al., 2019; Klys et al., 2018).”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “This example encapsulates an important issue of current explainable AI [1, 2].”
Explainable machine learning in deployment (2019, methodology): “Once the context and utility of the explanation are stated, understand what the stakeholder wants to do with the explanation [24].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, background): “[27] take a step further and define interpretability as a part of explainability.”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, background): “Machine learning research has been focusing more and more on interpretability (Gilpin et al., 2018).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, methodology): “To our knowledge, the term “faithful interpretability” was coined in Harrington et al. (1985), reinforced by Ribeiro et al. (2016), and is, we believe, most commonly used (e.g., Gilpin et al. (2018); Wu and Mooney (2018); Lakkaraju et al. (2019)).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “For XAI, see [1, 2, 31, 55, 61, 67, 75, 111, 120, 128], for causal and counterfactual inference, see [63, 76, 131, 145, 146, 159, 185], and for optimization, see [32, 137, 166].”
Understanding Multi-Head Attention in Abstractive Summarization (2019, background): “Gilpin et al. (2018) define an explanation to be an answer to “why questions” and consider it a trade-off between interpretability and completeness.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “It complements other explainable AI (XAI) surveys and critical retrospectives that focus on definitions, methods, and/or evaluation [33, 15, 77, 1, 103, 51, 42, 133, 26, 44, 82, 121, 12, 86, 54, 19], but not on datasets.”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Although there is no established consensus on the complete set of factors that makes an AI system explainable, XAI work commonly shares the goal of making an AI system’s functioning or decisions easy to understand by people [7, 14, 27, 31, 34, 62, 70, 82].”
SAT-based Decision Tree Learning for Large Data Sets (2021, background): “Since decision trees are easy to understand and interpret, they can contribute to the general objec-tive of explainable and interpretable AI (Gilpin et al. 2018).”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, methodology): “Moreover, the lack of interpretability on model prediction will raise the issue of trustworthy and fairness of sentiment classifiers in practice (Gilpin et al., 2018; Yu and Kumbier, 2019).”