Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

By Yunfeng Zhang, Q. Liao, Rachel K. E. Bellamy. 2020

984 citations recorded by the source corpus.

Today, AI is being increasingly used to help human experts make decisions in high-stakes scenarios. In these scenarios, full automation is often undesirable, not only due to the significance of the outcome, but also because human experts can draw on their domain knowledge complementary to the model's to ensure task success. We refer to these scenarios as AI-assisted decision making, where the individual strengths of the human and the AI come together to optimize the joint decision outcome. A key to their success is to appropriately calibrate human trust in the AI on a case-by-case basis; knowing when to trust or distrust the AI allows the human expert to appropriately apply their knowledge, improving decision outcomes in cases where the model is likely to perform poorly. This research conducts a case study of AI-assisted decision making in which humans and AI have comparable performance alone, and explores whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. Specifically, we study the effect of showing confidence score and local explanation for a particular prediction. Through two human experiments, we show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making, which may also depend on whether the human can bring in enough unique knowledge to complement the AI's errors. We also highlight the problems in using local explanation for AI-assisted decision making scenarios and invite the research community to explore new approaches to explainability for calibrating human trust in AI.

Subfields: Artificial Intelligence, Safety Research, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Social Psychology

Citations in the corpus

Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “For example, recent HCI studies repeatedly found that showing uncertainty information of individual predictions is more effective than local explanations to help people form appropriate reliance on AI [14, 105].”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “[3, 52] showed that confidence scores helped calibrate users’ trust, form a good mental model of the AI, and understand the error boundaries better.”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “AI assistance improves human performance compared to without any assistance; however, the performance of human-AI teams seldom surpasses AI alone in challenging prediction tasks [3, 5, 7, 20, 21, 32, 33, 37, 43, 54, 65, 66, 73].”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “Following recent HCI and FATE studies investigating how to design effective documentation interfaces (Crisan et al., 2022; Liao et al., 2023), we suggest that those designing model reports for LLMs should explore more interactive features.”
When combinations of humans and AI are useful: A systematic review and meta-analysis (2024, positive): “But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
To Trust or to Think (2021, background): “Thus, numerous studies have highlighted the importance of calibrated trust in improving AI-assisted decision-making [7, 9, 31, 44, 51, 53, 67].”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, background): “Prior work has shown that non-expert analysts benefit from interacting with machine learning models and their data [1, 19, 59].”
Semi-Automated Data Labeling (2020, result): “In a similar study, Zhang et al. (2020) measured improvement in users’ trust when provided with prediction and conﬁdence scores but found no signiﬁcant improvement in accuracy.”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, background): “While recent work has only begun to examine how uncertainty estimates might affect user interactions with ML models and decision task performance [7, 196], we highlight a few conclusions from the bulk”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “explanations have found popular XAI techniques to be ineffective [6, 80, 111], potentially risky [50, 95], and underused ar X iv :2 10 1.”
Trust in AI: progress, challenges, and future directions (2024, positive): “Another model, as described in Y. Zhang et al., (2020), compares the decisions made by AI systems and humans in their respective tasks to determine when to trust or distrust the AI.”
Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence (2021, background): “…the development of explainable AI systems is motivated by several social and ethical reasons (see, for example, [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]): increasing societal acceptance of prediction-based decisions, establishing trust in the results of these decisions, making algorithms…”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “Income category [83] Accuracy ↑ 65% 75% 73% ✗”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Third, perturbation-based methods, such as SHAP and LIME, manipulates parts of the inputs to generate explanations.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “In responding to this call, there is recently a growing line of literature on empirically evaluating the effectiveness of XAI methods (e.g., [14, 16, 41, 69, 71]).”
How do Data Science Workers Collaborate? Roles, Workflows, and Tools (2020, background): “, visualization [97]) and designs to make machine learning algorithms understandable by nontechnical users [3, 21, 39, 57, 102], whereas we argue for the explanation of decisions among the various data science creators of machine learning algorithms.”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “For instance, providing users with model conﬁdences can inﬂuence downstream decisions (Zhang et al., 2020), and users may want to adjust required conﬁdence thresholds on critical domains (e.g., health, safety, medicine).”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “For instance, Zhang et al. [222] explore the effect of intelligent system explanations in user trust calibration.”