Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems

By Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, Elena L. Glassman. International Conference on Intelligent User Interfaces, 2020

379 citations recorded by the source corpus.

Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decision-making tasks. Further, the subjective measures on evaluations with actual decision-making tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing human+AI teams that can reliably perform better than humans or AIs alone.

Subfields: Artificial Intelligence, Safety Research

View Paper: https://www.semanticscholar.org/paper/4244da5fc9b7e57ae7bb0c1b4c08c662af595a23

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Only until recently have researchers began to conduct controlled lab studies to rigorously evaluate popular XAI techniques [12, 13, 15, 22, 27, 53, 80], as well as studies to understand real-world user needs for AI explainability [43, 50, 58].”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, background): “decision-making, that is, to maximize the joint human-AI team performance [5, 6, 11].”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “[16] point out, performing proxy tasks in a controlled setting could induce a different cognitive process from a realistic setting, such as granting more attention to the explanations.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “AI assistance improves human performance compared to without any assistance; however, the performance of human-AI teams seldom surpasses AI alone in challenging prediction tasks [3, 5, 7, 20, 21, 32, 33, 37, 43, 54, 65, 66, 73].”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “It should be noted that although quantitative proxy metrics are necessary for an objective assessment of explanation quality and a formal comparison of explanation methods, they should be complemented with human evaluation methods before employing AI systems in real-life as good performance may not ”
To Trust or to Think (2021, background): “However, evidence suggests that explainable systems, also, have not had substantial success in reducing human overreliance on the AIs: when the AI suggests incorrect or suboptimal solutions, people still on average make poorer final decisions than they would have without AI’s assistance [2, 7, 30, 4”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, methodology): “We used a subset as the think aloud puts additional cognitive load on the participants and might "impact how people perform on cognitively-demanding tasks" [8].”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, methodology): “Since human-centered evaluations employ experiments with end-users or lay humans to assess the quality of explanations, subjective measures, such as trust and confidence, have been embraced as the focal point for the evaluation of explainable systems [33,64,65,69,92,93].”