Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

By Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, Eric Horvitz. 2019

344 citations recorded by the source corpus.

AI systems are being deployed to support human decision making in high-stakes domains such as healthcare and criminal justice. In many cases, the human and AI form a team, in which the human makes decisions after reviewing the AI’s inferences. A successful partnership requires that the human develops insights into the performance of the AI system, including its failures. We study the influence of updates to an AI system in this setting. While updates can increase the AI’s predictive performance, they may also lead to behavioral changes that are at odds with the user’s prior experiences and confidence in the AI’s inferences. We show that updates that increase AI performance may actually hurt team performance. We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams. Empirical results on three high-stakes classification tasks show that current machine learning algorithms do not produce compatible updates. We propose a re-training objective to improve the compatibility of an update by penalizing new errors. The objective offers full leverage of the performance/compatibility tradeoff across different datasets, enabling more compatible yet accurate updates.

Subfields: Artificial Intelligence, Health Informatics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

Citations in the corpus

Forward Compatible Few-Shot Class-Incremental Learning (2022, background): “They are introduced to the machine learning field in [5, 38].”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, background): “al [3] to describe the phenomenon of partial model regress in the context of human-AI collaboration.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “, positive or negative sentiment or binary recidivism prediction) [12, 13, 15, 16, 28, 39, 58, 62, 80, 81, 92, 94, 96, 107, 125, 137, 143, 152, 153, 155], multi-class, or categorical predictions (e.”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, methodology): “Based on results from Bansal et al. (2019), during model updates, deploy models whose error boundaries are b ackward compatible, i.e. by regularizing in order to minimize the introduction of new errors on instances where the user has learned to trust the system.”
No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (2020, neutral): “However, user feedback can have negative effects: decreased system performance [2, 65] or inconsistent mental models [9].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “ter. For papers with multiple domains or experiments, we took one sample with the most comparable human and AI performance. In this paper, we focus explicitly on AI-assisted decision making scenarios [3], where an AI assistant (e.g., a classification model) makes recommendations to a human (e.g., a ”
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020, background): “[2] to test mental modeling of error boundaries.”
Deep Class-Incremental Learning: A Survey (2023, background): “CIL with Bidirectional Compatibility: Compatibility is a design characteristic in software engineering community [240], [241], [242], [243], which was introduced to the machine learning community in [244], [245].”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, background): “[15] introduced the notion of compatibility of an AI update.”
Class-Incremental Learning: A Survey (2023, positive): “CIL with Bidirectional Compatibility : Compatibility is a design characteristic in software engineering community [229], [230], [231], [232], which was introduced to the machine learning community in [233], [234].”