How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation

By Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Samuel J. Gershman, Finale Doshi-Velez. 2018

262 citations recorded by the source corpus.

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable in the specific context of verification. Suppose we have a machine learning system that predicts X, and we provide rationale for this prediction X. Given an input, an explanation, and an output, is the output consistent with the input and the supposed rationale? Via a series of user-studies, we identify what kinds of increases in complexity have the greatest effect on the time it takes for humans to verify the rationale, and which seem relatively insensitive.

Subfields: Artificial Intelligence, Materials Chemistry, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, studying the efficacy of interpretability tools or exploring possible interfaces–would be welcome, as would further exploration of how practitioners currently use such tools [81] [82] [83] [78] [84].”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “Lastly, because it is not yet clear what makes explanations good for humans [35], one should extend explanatory interactions towards using alternative explanations, multiple modalities and counterfactuals [36, 37].”
Deep Weighted Averaging Classifiers (2018, background): “As emphasized by papers that have made use of user studies [16, 22, 36], careful empirical work is required to evaluate the effectiveness of explanations, and we leave such an evaluation of this approach for future work.”
Do explanations make VQA models more predictable to a human? (2018, background): “or helping them achieve an end goal (Narayanan et al., 2018; Kulesza et al., 2012).”
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey (2019, background): “Another approach towards understanding is to evaluate the human-interpretability of explanation [107].”
The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, background): “And despite widespread calls for explanations and interpretable models, recent studies have found that simple models do not lead to better human performance than black box models [62] and that varying algorithmic explanations does not a ect human accuracy [61].”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, background): “The question of evaluating and comparing explanations becomes even more complex when integrating human factors such as interpretability, manageability, and overall utility of the XAI system [116], [105].”
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models (2018, unknown): “8 RELATED WORK Various methods [4, 34] have been proposed to generate explanations for deep learning model predictions.”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “However, despite recent effort toward a scientific understanding of human-AI interaction [29, 68, 103], XAI research still struggles with a lack of understanding of real-world user needs for AI transparency, and by far little consideration of what practitioners need to create explainable AI products”
Explainable Automated Fact-Checking: A Survey (2020, background): “, 2016) and SHAP (Lundberg and Lee, 2017); work on evaluating explanations (Narayanan et al., 2018); and the formalization of desiderata (i.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “[2, 111, 136, 146]) argue that relying on such anecdotal evidence alone is insufficient and that other aspects of the explanations should be evaluated as well.”
Measuring the Quality of Explanations: The System Causability Scale (SCS) (2019, background): “Another approach measures a proxy for utility such as simplicity [11, 23] or response time in an application [24].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “n natural language (Section3.5). Admittedly, explanations can also be presented in other ways. For structured data, most studies present the salient features, sometimes paired with importance weights [8, 21, 47, 52, 63]. On vision tasks, some work takes the demonstrate-by-example approach, i.e., by ”
Expl(AI)n It to Me – Explainable AI and Information Systems Research (2021, background): “So far, the majority of previous studies has primarily focused on how people respond to different types of explanations, subjectively measured intuitiveness and usability of specific interpretability methods, and whether the model interpretability can improve the performance of human decision making”