Definitions, methods, and applications in interpretable machine learning

By W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu. 2019

1,811 citations recorded by the source corpus.

Significance The recent surge in interpretability research has led to confusion on numerous fronts. In particular, it is unclear what it means to be interpretable and how to select, evaluate, or even discuss methods for producing interpretations of machine-learning models. We aim to clarify these concerns by defining interpretable machine learning and constructing a unifying framework for existing methods which highlights the underappreciated role played by human audiences. Within this framework, methods are organized into 2 classes: model based and post hoc. To provide guidance in selecting and evaluating interpretation methods, we introduce 3 desiderata: predictive accuracy, descriptive accuracy, and relevancy. Using our framework, we review existing work, grounded in real-world studies which exemplify our desiderata, and suggest directions for future work. Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the predictive, descriptive, relevant (PDR) framework for discussing interpretations. The PDR framework provides 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post hoc categories, with subgroups including sparsity, modularity, and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often underappreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “We hope the work here will help push the field towards a more rigorous way to use interpretability methods, a point which will become increasingly important as interpretable machine learning develops as a field (Doshi-Velez & Kim, 2017; Murdoch et al., 2019).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “Sometimes referred to as how “trustworthy” (Camburu et al., 2019) or “descriptive” (Carmona et al., 2015; Biecek, 2018) the interpretation is, or as “descriptive accuracy” (Murdoch et al., 2019).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “, 2019), or a combination of these topics (Lipton, 2018; Doshi-Velez and Kim, 2017; Adadi and Berrada, 2018; Murdoch et al., 2019; Verma et al., 2020; Burkart and Huber, 2021), but not on datasets.”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “The effectiveness and adequacy of explainability as a tool for AI safety may be limited in specific scenarios [269].”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “Murdoch, Singh, Kumbier, Abbasi-Asl and Yu [19] present a common vocabulary to help select and evaluate explainable AI methods.”
Interpretable Random Forests via Rule Extraction (2020, background): “In particular, Murdoch et al. (2019) introduce the notion of simulatable models when a human is able to reproduce the prediction process by hand.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Those terms, together with explainability and interpretability (and to some extent also intelligibility) are typically used interchangeably [29, 32, 93, 177], although a few distinguish between them [21, 173].”
Explainable AI: A Review of Machine Learning Interpretability Methods (2020, background): “[20] published a survey in 2019, in which they created an interpretability framework in the hope that it would help to bridge the aforementioned gap in the field.”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “Research-wise, the field is maturing in terms of methods surveys [75,41,120,96,1,6,23,15], further consolidation of terms and knowledge [42,22,82,97,88,17], and work about defining interpretability or evaluation of IML methods [74,73,95,49].”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, background): “Similarly, the examples in [98] also mostly focus on visual domains and explanations.”
Born-Again Tree Ensembles (2020, background): “Research in this domain has signiﬁcantly increased (Murdoch et al., 2019) with numerous works focusing on the construction of optimal sparse trees (Hu Formally, let ( X , y ) = { x i , y i } ni =1 be a training set in which each x i ∈ R p is a p -dimensional numerical feature vector, and each y i ∈…”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “The work of Murdoch et al. (2019) introduces the Predicitve, Descriptive, Relevant (PDR) framework.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “It complements other explainable AI (XAI) surveys and critical retrospectives that focus on definitions, methods, and/or evaluation [33, 15, 77, 1, 103, 51, 42, 133, 26, 44, 82, 121, 12, 86, 54, 19], but not on datasets.”