Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems

By Rafal Kocielnik, Saleema Amershi, Paul N. Bennett. 2019

363 citations recorded by the source corpus.

AI technologies have been incorporated into many end-user applications. However, expectations of the capabilities of such systems vary among people. Furthermore, bloated expectations have been identified as negatively affecting perception and acceptance of such systems. Although the intelligibility of ML algorithms has been well studied, there has been little work on methods for setting appropriate expectations before the initial use of an AI-based system. In this work, we use a Scheduling Assistant - an AI system for automated meeting request detection in free-text email - to study the impact of several methods of expectation setting. We explore two versions of this system with the same 50% level of accuracy of the AI component but each designed with a different focus on the types of errors to avoid (avoiding False Positives vs. False Negatives). We show that such different focus can lead to vastly different subjective perceptions of accuracy and acceptance. Further, we design expectation adjustment techniques that prepare users for AI imperfections and result in a significant increase in acceptance.

Subfields: Computer Science Applications, Information Systems and Management, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems and Management

In Social Psychology

Citations in the corpus

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “Understanding what the AI does not know is also extremely important for creating a suitable mental model of the AI’s capabilities.(21,28,29) To do this, the AI system must be able to estimate the uncertainty in its outputs.”
Manipulating and Measuring Model Interpretability (2018, background): “They and others [55] also found that people become more willing to use computational systems when they are given the opportunity to review and potentially modify the systems, even when the modifications have no effects [104].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “[74] use Wizard of Oz to generate a table of representative instances for each prediction class respectively to help user understand how the AI component operates.”
Machine Learning Operations (MLOps): Overview, Definition, and Architecture (2022, background): “However, the success of many productive ML applications in real-world settings falls short of expectations [21].”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, background): “Research in the intersection of AI and HCI has found that interaction improves when setting expectations right about what the system can do and how well it performs (Kocielnik, Amershi, and Bennett 2019; Amershi et al. 2019).”
Trustworthy Artificial Intelligence: A Review (2022, background): “Before-the-loop 1 [82, 113, 208] Before-the-loop, Over-the-loop 1, 4 [82, 204] In-the-loop, Over-the-loop 3, 4 [103, 168, 203]”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “Prior studies have examined how users interact with imperfect AI [11, 23, 26–28, 35, 37, 45].”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “of model explanations in an explanation group. Examples for the baseline include approaches that remove model explanations related components and features form the interface in the baseline condition [103, 160]. In other work, Poursabzi et al. [168] also included a no AI baseline to measure particip”