Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

By Peter Hase, Mohit Bansal. 2020

374 citations recorded by the source corpus.

Algorithmic approaches to interpreting machine learning models have proliferated in recent years. We carry out human subject tests that are the first of their kind to isolate the effect of algorithmic explanations on a key aspect of model interpretability, simulatability, while avoiding important confounding experimental factors. A model is simulatable when a person can predict its behavior on new inputs. Through two kinds of simulation tests involving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Composite approach that combines explanations from each method. Clear evidence of method effectiveness is found in very few cases: LIME improves simulatability in tabular classification, and our Prototype method is effective in counterfactual simulation tests. We also collect subjective ratings of explanations, but we do not find that ratings are predictive of how helpful explanations are. Our results provide the first reliable and comprehensive estimates of how explanations influence simulatability across a variety of explanation methods and data domains. We show that (1) we need to be careful about the metrics we use to evaluate explanation methods, and (2) there is significant room for improvement in current methods.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “In this setup, humans are presented with an explanation and an input, and must correctly predict the model’s output (Doshi-Velez and Kim, 2017; Lertvittayakumjorn and Toni, 2019; Hase and Bansal, 2020).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “Finally, users simulated what the model would do on six counterfactuals (Hase and Bansal, 2020), two from each condition (in random order).”
PRover: Proof Generation for Interpretable Reasoning over Rules (2020, background): “Several approaches have focused on formalizing the notion of interpretability (Rudin, 2019; Doshi-Velez and Kim, 2017; Hase and Bansal, 2020), tweaking features for local model interpretability (Ribeiro et al.”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, methodology): “ipation’ While not an evaluation of trust, there are methods of evaluating the ability of users to successfully anticipate the AI’s behavior, an important aspect of Human-AI trust, via simulatability [15, 25]—the ability of the user to simulate the outcome of the AI on an instance level. Thus, simul”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “A recent study trained students with 16 or 32 examples, and tested if students could simulate the model better using different explanations, however the observed differences among techniques were not statistically significant (Hase and Bansal, 2020).”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “While many studies showed positive results that XAI techniques can improve people’s understanding of models [46, 59, 68, 87], in this section we draw attention to a few pitfalls of XAI based on recent HCI research.”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “Hase and Bansal (2020) further refines simulatability evaluation by defining testing regimen where it can be properly assessed without false signal.”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “We chose text classification because it is a popular task in natural language processing (NLP) that has been used in several previous studies on human-AI teaming [17, 24, 35, 43, 53, 70], requires little domain expertise, and is thus amenable to crowdsourcing.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “Those terms, together with explainability and interpretability (and to some extent also intelligibility) are typically used interchangeably [29, 32, 93, 177], although a few distinguish between them [21, 173].”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, methodology): “Simulatability-based evaluations have been conducted before (Ribeiro et al., 2018; Hase and Bansal, 2020), but we are the ﬁrst to consider NL explanations and employ model-based controls for label leakage.”
Natural Language Deduction through Search over Statement Compositions (2022, background): “This limitation prevents users from understanding and accommodating models’ affordances (Hase and Bansal, 2020; Bansal et al., 2021).”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, methodology): “Highlight explanations have commonly been used in prior XAI studies [4, 26, 33, 53, 71, 97].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision (2020, background): “ford et al. 2019) and BERT (Devlin et al. 2Humans also post-hoc rationalize decisions, and it is known to be ﬂawed (Gazzaniga and LeDoux 2013). For recent works discussing rationale faithfulness, see Hase and Bansal (2020), Jacovi and Goldberg (2020), and Wiegreffe, Sarah and Marasovic, Ana,´ and Sm”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “8 Plausibility is also evaluated with forward simulation (Hase and Bansal, 2020): “given an input and an ‘explanation,’ users must predict what a model would output for the given input”.”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, methodology): “…Structured Explanations: There is useful previous work on developing interpretable and explain-able models (Doshi-Velez and Kim, 2017; Rudin, 2019; Hase and Bansal, 2020; Jacovi and Goldberg, 2020) for NLP. Explanations in NLP take three major forms – (1) extractive rationales or highlights…”