Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance

By Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, Eric Horvitz. 2019

564 citations recorded by the source corpus.

Decisions made by human-AI teams (e.g., AI-advised humans) are increasingly common in high-stakes domains such as healthcare, criminal justice, and finance. Achieving high team performance depends on more than just the accuracy of the AI system: Since the human and the AI may have different expertise, the highest team performance is often reached when they both know how and when to complement one another. We focus on a factor that is crucial to supporting such complementary: the human’s mental model of the AI capabilities, specifically the AI system’s error boundary (i.e. knowing “When does the AI err?”). Awareness of this lets the human decide when to accept or override the AI’s recommendation. We highlight two key properties of an AI’s error boundary, parsimony and stochasticity, and a property of the task, dimensionality. We show experimentally how these properties affect humans’ mental models of AI capabilities and the resulting team performance. We connect our evaluations to related work and propose goals, beyond accuracy, that merit consideration during model selection and optimization to improve overall human-AI team performance.

Subfields: Artificial Intelligence, Management Information Systems, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Safety Research

Citations in the corpus

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “Understanding what the AI does not know is also extremely important for creating a suitable mental model of the AI’s capabilities.(21,28,29) To do this, the AI system must be able to estimate the uncertainty in its outputs.”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “Currently, there are no standardised metrics for characterising the goodness-of-data [11, 13]; research on metrics is emerging [15, 91] but not yet widely adopted in AI system building.”
Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021, background): “For optimal results, the human needs to understand when to listen to the computer and when to exercise their own agency [3].”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, methodology): “• Studies using actual tasks to evaluate the performance of human and the system on the ML-informed decision-making tasks [68,69].”
To Trust or to Think (2021, background): “A leading view to explain the inferior team performance is that humans overtrust the AI [1].”
Harnessing Explanations to Bridge AI and Humans (2020, background): “(2019) discuss as a general hurdle in human-AI collaboration [2].”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, methodology): “User Understanding of Model Interview ([40]) and Self-explanation ([20, 47, 164]) Likert-scale Questionnaire ([99, 111, 113, 124, 133, 171]) Model Output Prediction User Prediction of Model Output ([96, 172, 173]) Model Failure Prediction User Prediction of Model Failure ([15, 161])”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “Prior XAI literature has used a variety of tasks, including binary classification [3, 4, 8, 54] and question answering about images [12, 15] and text [4, 26].”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, methodology): “Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task [3, 17, 23, 46].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “” While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5], it more importantly shows how the AI’s confdence is a simple, yet salient feature via which users create a mental model of the AI’s global behavior [23].”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, result): “[2] but much smaller than deception detection, and the out-of-distribution performance gap between human alone and AI alone becomes similar to LSAT in Bansal et al.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “We treated the type of explanation a participant received as the fixed effect, the participant as the random effect, and controlled for the participant’s age, gender, and education as covariates 5 .”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “Artificial Alien medicine recommendation [79, 104], alien recipe recommendation [79, 104], jellybean counting [109], broken glass prediction [153], defective object pipeline [11, 12], news reading time prediction [134], water pipe failure prediction [10], math questions [45].”