An Evaluation of the Human-Interpretability of Explanation

By Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel J. Gershman, Finale Doshi-Velez. 2019

186 citations recorded by the source corpus.

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable under three specific tasks that users may perform with machine learning systems: simulation of the response, verification of a suggested response, and determining whether the correctness of a suggested response changes under a change to the inputs. Through carefully controlled human-subject experiments, we identify regularizers that can be used to optimize for the interpretability of machine learning systems. Our results show that the type of complexity matters: cognitive chunks (newly defined concepts) affect performance more than variable repetitions, and these trends are consistent across tasks and domains. This suggests that there may exist some common design principles for explanation systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “variables, [61] finds out that interpretability of explanations in the form of rule sets is most affected by cognitive chunks, explanation size and little effected by variable repetition.”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “…the context of three tasks they are likely to perform given an explanation of an ML system: (i) predicting the system’s output, (ii) verifying whether the output is consistent with the explanation, and (iii) determining if and how the output would change if we change the input (Lage et al. 2019).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, neutral): ““Readability” and “plausibility” are also referred to as “human-interpretability” and “persuasive-ness”, respectively (e.g., Lage et al. (2019); Herman (2017)).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Specifically in the context of XAI, it has been shown that counterfactuals can “make the decisions of inscrutable systems intelligible to developers and users” [35], and that people perform better at predicting model behavior when presented with counterfactual instances [98].”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “Qualitative metrics include asking about the usefulness of, satisfaction with, and trust in provided explanations by means of interviews or questionnaires [61, 62].”
Reasoning Models Don't Always Say What They Think (2025, unknown): “Plausibility evaluates the factual correctness of a CoT and how convincingly it justifies the model’s response (Herman, 2017; Lage et al., 2019; Jacovi and Goldberg, 2020).”
Evaluating and Characterizing Human Rationales (2020, unknown): “ERASER (DeYoung et al., 2019) includes three additional datasets: CoS-( Lai and Tan, 2019; Poursabzi-Sangdeh et al., 2018; Lage et al., 2018; Lai et al., 2020).”
Algorithmic Recourse: from Counterfactual Explanations to Interventions (2020, background): “shown that designers of interpretable machine learning systems use counterfactual explanations for predicting model behavior [24] or uncovering inaccuracies in the data profile of individuals [50].”
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (2019, background): “Recent work explores multi-agent reinforcement learning in repeated social interactions to begin to identify conditions on the environment and the agents’ cognitive capacities which lead to the emergence of cooperation (see [39,43] and related blog entries).”
Towards Explainable Artificial Intelligence (2019, background): “Some works (e.g., [43]) have already started to investigate human factors in explainable AI.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “Typical methods include ask people to rank the input data features based on their inuence to overall predictions [18, 33], to indicate the direction of change in the model’s prediction when a feature’s value is altered [16, 18, 29], to simulate the model’s predictions [16, 24, 40, 47, 59], to answer”
Human Evaluation of Models Built for Interpretability (2019, methodology): “See our tech report for details (Lage et al. 2019).”
Harnessing Explanations to Bridge AI and Humans (2020, background): “Some experiments in this vein have included predicting apartment prices [13], detecting deceptive online reviews [11], assessing social media toxicity [3], performing various artificial tasks [9], and recidivism prediction [5].”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, methodology): “…and can have a straightforward application for digital healthcare, for example, decision sets or rule sets have been studied for interpretability [159] and Model Understanding through Subspace Explanations (MUSE) method [160] has been developed to describe the projections of the global model by…”
Evaluating XAI: A comparison of rule-based and example-based explanations (2021, unknown): “Several XAI methods try to identify examples to generate such explanations, for example those from [43–47].”
The explanation game: a formal framework for interpretable machine learning (2020, unknown): “, 2017) – the statistical procedure that produces rule lists – and a growing number of psychological studies suggests that users find such explanations especially intelligible (Lage et al., 2018).”