Metrics for Explainable AI: Challenges and Prospects

By Robert R. Hoffman, Shane T. Mueller, Gary Klein, Jordan Litman. 2018

907 citations recorded by the source corpus.

The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We discuss specific methods for evaluating: (1) the goodness of explanations, (2) whether users are satisfied by explanations, (3) how well users understand the AI systems, (4) how curiosity motivates the search for explanations, (5) whether the user's trust and reliance on the AI are appropriate, and finally, (6) how the human-XAI work system performs. The recommendations we present derive from our integration of extensive research literatures and our own psychometric evaluations.

Subfields: Artificial Intelligence, Health Informatics, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Some attempts have been done recently around the measurement of XAI, as reviewed thoroughly in [349,350] .”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “Human-based evaluation is a key step in validating the utility of tools for ML explainability (Hoffman et al., 2018).”
In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes (2019, methodology): “[18], some of these facets are problematic when applied to a machine.”
Sequential Explanations with Mental Model-Based Policies (2020, methodology): “…providing the current explanation, as observations of the mental model (i.e., the context): explainee’s satisfaction with the prior explanation (Hoffman et al., 2018; Madumal et al., 2019) and explainee’s local simulatability of the black-box model for each classiﬁcation possibility (Hoffman…”
Explainability in Deep Reinforcement Learning (2020, background): “We also present the level of the provided explanation ( local if it explains only predictions, global if it explains the whole model), and the audience concerned by the explanation, as discussed in Section 1.1.”
Cross-Loss Influence Functions to Explain Deep Network Representations (2020, background): “…have been proposed, including interpretable neural network approaches (Wang et al., 2016), natural language generation of explanations (Mullenbach et al., 2018; Wiegreﬀe and Pinter, 2019; Chen et al., 2021), and many others Hoﬀman et al. (2018); Holzinger et al. (2020); Linardatos et al. (2021).”
Causability and explainability of artificial intelligence in medicine (2019, background): “We conclude with pointing to the urgent need of a systems causability scale to measure the quality of an explanation (Hoffman, Mueller, Klein, & Litman, 2018), which must also include social aspects of human communication (Miller, 2019).”
A historical perspective of explainable Artificial Intelligence (2020, background): “…the criteria for a good explanation as of today is an active debate in various fields, including cognitive science, computer science, psychology, and philosophy (Confalonieri et al., 2019; Guidotti et al., 2018; Hoffman, Mueller, Klein, & Litman, 2018; Lipton, 2018; Lombrozo, 2016; Miller, 2019).”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, background): “[19] proposed metrics for explainable systems that are grounded in the subjective evaluation of a system (e.”
Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation (2023, background): “WHY is it important for trustworthiness? The required auditability property of Trustworthy AI systems demands the development of practical tools [106] that are capable of verifying desirable properties of neural networks such as stability, sensitivity, relevance or reachability [107], as well as met”
Trustworthy AI: From Principles to Practices (2021, background): “, user satisfaction, mental models, and trust [144, 155, 267].”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “In the work of Hoffman et al. (2018), each domain expert had the possibility to rate explanations by a scoring system.”
Human-Centered AI (2021, background): “This means that humans need to be able to understand the AI sufficiently well to predict system behavior [16] .”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, methodology): “End users may only become aware of these inconsistencies when they recall their abstractions to self-explain their understanding of the global ML model behavior [22].”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “Trustworthiness Enabling positive user attitude toward the system that emerges from knowledge, experience, and emotion [82, 85].”
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020, background): “[14] recognizes that forming a good mental model of AI is the key to effectively appropriating trust and usage.”
Six Human-Centered Artificial Intelligence Grand Challenges (2023, background): “People are responsible and accountable for the system design and behavior which requires them to comprehend AI well enough to anticipate potentially problematic system behavior (Hoffman et al., 2018).”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, background): “For example, Hoffman et al. [74] presented metrics for explainable systems that are grounded in the subjective evaluation of a system (e.g., user trust, satisfaction, and understanding).”