Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making

By Xinru Wang, Ming Yin. International Conference on Intelligent User Interfaces, 2021

418 citations recorded by the source corpus.

This paper contributes to the growing literature in empirical evaluation of explainable AI (XAI) methods by presenting a comparison on the effects of a set of established XAI methods in AI-assisted decision making. Specifically, based on our review of previous literature, we highlight three desirable properties that ideal AI explanations should satisfy—improve people’s understanding of the AI model, help people recognize the model uncertainty, and support people’s calibrated trust in the model. Through randomized controlled experiments, we evaluate whether four types of common model-agnostic explainable AI methods satisfy these properties on two types of decision making tasks where people perceive themselves as having different levels of domain expertise in (i.e., recidivism prediction and forest cover prediction). Our results show that the effects of AI explanations are largely different on decision making tasks where people have varying levels of domain expertise in, and many AI explanations do not satisfy any of the desirable properties for tasks that people have little domain expertise in. Further, for decision making tasks that people are more knowledgeable, feature contribution explanation is shown to satisfy more desiderata of AI explanations, while the explanation that is considered to resemble how human explain decisions (i.e., counterfactual explanation) does not seem to improve calibrated trust. We conclude by discussing the implications of our study for improving the design of XAI methods to better support human decision making.

Subfields: Artificial Intelligence, Management Information Systems, Safety Research

View Paper: https://www.semanticscholar.org/paper/c79ba865574cb9635dbca135cec99ee56125778c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Safety Research

Citations in the corpus

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…a wave of HCI studies repeatedly showed that AI explanations can lead to overreliance—increasing people’s tendency to mistakenly follow the AI outputs even when they are wrong (Bansal et al., 2021; V. Chen et al., 2023; Poursabzi-Sangdeh et al., 2021; X. Wang and Yin, 2021; Y. Zhang et al., 2020).”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “Such tests to assess people’s understanding without a specific end goal can fail to predict the success of using an XAI technique in real tasks that people seek explanations for, such as debugging models [54] or improving decision-making [16, 100].”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “Future work should explore our theory on high-stakes tasks such as recidivism prediction [92], where the consequences of a wrong decision are much higher.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “AI assistance improves human performance compared to without any assistance; however, the performance of human-AI teams seldom surpasses AI alone in challenging prediction tasks [3, 5, 7, 20, 21, 32, 33, 37, 43, 54, 65, 66, 73].”