Manipulating and Measuring Model Interpretability

By Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, Hanna M. Wallach. 2018

848 citations recorded by the source corpus.

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model’s predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N = 3, 800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model’s predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model’s sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.

Subfields: Artificial Intelligence, General Decision Sciences, Management Science and Operations Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Decision Sciences

In Management Science and Operations Research

In Safety, Robustness & Interpretability

Citations in the corpus

Measurement and Fairness (2019, background): “Indeed, interacting with measurements can create room for more individual judgments, automation bias, and (lack of) intuition and interpretability [6, 16, 28, 38, 55, 60].”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “…has a rich history in machine learning inter-pretability as a reliable measure of rationale quality from the lens of utility to an end-user (Kim et al., 2016; Chandrasekaran et al., 2018; Hase and Bansal, 2020; Yeung et al., 2020; Poursabzi-Sangdeh et al., 2021; Rajagopal et al., 2021, i.a. ).”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “[141, 237], have performed extensive human experiments on interpretable models and posthoc explanations of black box models, with”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “Although important studies from the HCI community have probed these questions, 67,86,87 more collaborative work between AI and HCI researchers, as well as statisticians and others experienced in communicating about uncertainty, will be crucial for focusing technical research toward developing…”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “This is already the case in regulated domains such as finance [78].”
Explainable machine learning in deployment (2019, background): “• Static Consumption: Will the explanation be used as a oneoff sanity check for some stakeholders or shown to other stakeholders as reasoning for a particular prediction [48]? • Dynamic Model Updates: Will the explanation be used to garner feedback from the stakeholder as to how themodel ought to be”
Challenges of Real-World Reinforcement Learning (2019, background): “For explainability, there is not a modiﬁcation to the domain, but evaluation must be performed qualititatively with humans determining the quality of the explainability (e.g. perhaps with mechanical turk ( Poursabzi-Sangdeh et al., 2018)).”
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “While a substantial body of work examines user information needs at prediction time [12, 60, 63], or how to design AI assistance to integrate seamlessly with practices [78], we focus on the initial human–AI onboarding phase , when users are first being introduced to an AI system, learning its…”
Interpretable Machine Learning (2019, background): “There is also a risk that oversimplified explanations could cause users to develop a false sense of understanding of a system and over-rely on its outputs.(105,106)”
From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices (2019, background): “The next observation of note is that few of the available tools surveyed provide meaningful ways to assess, and respond to, the impact that the data-processing involved in their ML algorithm has on an individual, and even less on the impact on society as a whole (Poursabzi-Sangdeh et al. 2018).”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “Instead of having one groundtruth explanation, various quantiﬁable aspects of interpretability are emerging [87,86,77,46,131,3,102,87,21,31].”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, methodology): “2) has been informed by prior work related to measuring and improving model interpretability [45, 55], and proposals for contrastive explanations [45] are particularly aligned with our approach to interactivity and seeding the IMC with examples (as described in Section 4.”
Debugging Tests for Model Explanations (2020, background): “Poursabzi-Sangdeh et al. [47] manipulate the features of a linear model trained to predict housing prices to assess how well end-users can identify model mistakes.”
Transparency and trust in artificial intelligence systems (2020, background): “Multiple studies found that trust into assistive AI technology can be increased most effectively by confronting users with an easy to understand explanation of why a specific prediction was made (Herlocker et al., 2000; Poursabzi-Sangdeh et al., 2018; Zhao et al., 2019).”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, background): “accuracy, response time needed, likelihood to deviate, or ability to detect errors [3,66], and even physiological responses from humans during experimental tasks [67].”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “examining how useful an explanation is with human users [24, 59].”