Explaining Explanations in AI

By Brent Daniel Mittelstadt, Chris Russell, Sandra Wachter. 2018

780 citations recorded by the source corpus.

Recent work on interpretability in machine learning and AI has focused on the building of simplified models that approximate the true criteria used to make decisions. These models are a useful pedagogical device for teaching trained professionals how to predict what decisions will be made by the complex system, and most importantly how the system might break. However, when considering any such model it's important to remember Box's maxim that "All models are wrong but some are useful." We focus on the distinction between these models and explanations in philosophy and sociology. These models can be understood as a "do it yourself kit" for explanations, allowing a practitioner to directly answer "what if questions" or generate contrastive explanations without external assistance. Although a valuable ability, giving these models as explanations appears more difficult than necessary, and other forms of explanation may not have the same trade-offs. We contrast the different schools of thought on what makes an explanation, and suggest that machine learning might benefit from viewing the problem more broadly.

Subfields: Artificial Intelligence, History and Philosophy of Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In History and Philosophy of Science

In Safety, Robustness & Interpretability

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “Contrastive Natural Language Explanations Existing research in social science and cognitive science (Miller, 2019; Mittelstadt et al., 2019) suggests contrastive explanations are more effective in human learning than descriptive explanations.”
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning (2019, background): “Counterfactual reasoning has been identiﬁed as a particularly effective way to present explanations of the decision boundaries of deep models (Mittelstadt et al., 2019).”
Explainable machine learning in deployment (2019, background): “Contrastive explanations that highlight contextually relevant information to the model output are most similar to human explanations[40]; however, in their current form, finding the relevant set of plausible counterfactual point is no clear.”
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods (2019, background): “However, instances generated using such perturbations could potentially be off-manifold or out-of-distribution (OOD) [16].”
Efficient Search for Diverse Coherent Explanations (2019, background): “For a more detailed review of the literature, please see Mittelstadt et al. (2019).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Communicating the domain of applicability to users is critical for building trust [127, 177].”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Moreover, XAI’s disconnect with the philosophical and psychological grounds of human explanations has been duly noted [71], as best represented by Miller’s call for leveraging insights from the Social Sciences [70].”
Artificial intelligence in clinical and genomic diagnostics (2019, background): “Interpretable machine-learning methods are an active area of computer science research [87], but most interpretable AI approaches involve the production of a simplified and potentially inaccurate approximation of the more complex AI system [86].”
Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence (2021, background): “The argumentative nature of the explanations generated by AI systems has been described by [23, 51], among others.”
"The human body is a black box": supporting clinical decision-making with deep learning (2019, background): “What should constitute an explainable algorithm in clinical practice when the definition and underlying pathophysiology of sepsis are incompletely understood in the first place? Finally, there remains a lack of consensus around what should or could constitute “interpretable” or “explainable” AI [13,”
What Do We Want From Explainable Artificial Intelligence (XAI)? - A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research (2021, background): “Crucial desiderata of affected parties are fairness [7, 9, 11, 122] and morality/ethics [6, 11, 127].”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, methodology): “In an effort to mitigate the risks for harm in AI/ML systems, researchers and developers have proposed strategies to make systems more explainable, transparent, and contestable (e.g., [3, 24, 34, 41, 45, 47, 74]).”
Bridging the Gap Between Ethics and Practice (2020, background): “This controversial GDPR requirement is vague and difficult to satisfy in general, but international research efforts at developing Explainable AI (XAI) have blossomed [9, 47, 48, 106, 116, 75, 71].”
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2022, background): “There is a growing body of research in explainable AI (Mittelstadt et al., 2019).”
The explanation game: a formal framework for interpretable machine learning (2020, neutral): “The importance of contrastive explanation is highlighted by several philosophers (Hitchcock 1999; Potochnik 2015; van Fraassen 1980), and has recently begun to receive attention in the iML literature as well (Miller 2019; Mittelstadt et al. 2019).”