Understanding the Effect of Accuracy on Trust in Machine Learning Models

By Ming Yin, Jennifer Wortman Vaughan, Hanna M. Wallach. International Conference on Human Factors in Computing Systems, 2019

599 citations recorded by the source corpus.

We address a relatively under-explored aspect of human-computer interaction: people's abilities to understand the relationship between a machine learning model's stated performance on held-out data and its expected performance post deployment. We conduct large-scale, randomized human-subject experiments to examine whether laypeople's trust in a model, measured in terms of both the frequency with which they revise their predictions to match those of the model and their self-reported levels of trust in the model, varies depending on the model's stated accuracy on held-out data and on its observed accuracy in practice. We find that people's trust in a model is affected by both its stated accuracy and its observed accuracy, and that the effect of stated accuracy can change depending on the observed accuracy. Our work relates to recent research on interpretable machine learning, but moves beyond the typical focus on model internals, exploring a different component of the machine learning pipeline.

Subfields: Artificial Intelligence, Safety Research

View Paper: https://www.semanticscholar.org/paper/b4b1cbd74029f46ef9b462290a46111217552761

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “[214] studied the importance of model accuracy on user trust.”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “[72] investigate the effect of observed and stated accuracy on humans’ trust in AI and find that both stated and observed accuracy can affect human trust in AI.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “Following earlier literature [70, 71], we quantified people’s reliance on the model using the fraction of tasks in which the participant’s final prediction was the same as the model’s prediction (i.e., agreement fraction ).”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, methodology): “Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task [3, 17, 23, 46].”
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “Relatedly, others have recently studied whether laypeople’s trust in a model varies depending on the model’s stated accuracy and how it differs from observed accuracy in practice [79].”
To Trust or to Think (2021, unknown): “This condition captures another common approach for designing interfaces for human-AI collaboration [2, 40, 66].”
Teachers' trust in AI-powered educational technology and a professional development program to improve it (2022, methodology): “To establish initial positive expectations, and in line with (Yin et al., 2019), we presented the teachers with the accuracy of an ML automated grading system that was trained and tested on a large corpus (two open-ended questions, ~ 1300 constructed responses from ~ 650 students, ~ 85% accuracy).”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, background): “…is informative for decision-making, research in human-centered machine learning (Gillies et al. 2016) and HCI shows that people have difficulties with correctly interpreting probabilistic statements (Handmer and Proudley 2007) and even system accuracy itself (Yin, Vaughan, and Wallach 2019).”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “, positive or negative sentiment or binary recidivism prediction) [12, 13, 15, 16, 28, 39, 58, 62, 80, 81, 92, 94, 96, 107, 125, 137, 143, 152, 153, 155], multi-class, or categorical predictions (e.”
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference (2019, background): “While previous studies have shown that explainability methods can mediate in an agency-efﬁciency trade-off [53,88], none of them demonstrate a way for users to retain agency while gaining the efﬁciency beneﬁts of models interaction.”
No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (2020, background): “, observed and stated model accuracy can affect users’ trust [66]), we retained the Study 1 hypotheses regarding our primary measures (frustration and expected change).”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “(4) AI design: Although increasing benefits of the following may result in increases in overreliance, designers are also able to manipulate (1) perceived model accuracy [95], (2) AI anthropromorphization [50], and (3) the competence associated with metaphors describing the system [44].”
Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021, background): “They’ve investigated factors including reported model confidence, model explanations, overall model accuracy, and initial model accuracy [34, 37, 53, 54].”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “…groups like data scientists and model developers, business decision-makers, regulators and auditors, end-users, and impacted groups (i.e., the people who are directly or indirectly affected by a model or application) (Hong et al., 2020; Liao and Varshney, 2021; Vaughan and Wallach, 2021).”
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020, methodology): “…while research repeatedly demonstrates that providing high-performance indicators of an AI system, such as showing high accuracy scores, could enhance people’s trust and acceptance of the system [17, 30, 32], they may not help people distinguish cases they can trust from those they should not.”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, background): “There was a common hesitation among informants to present ML performance metrics such as accuracy, not only because a numerical value could be hard to interpret by lay users, but also there may be discrepancy between performances on the test data and the actual data, creating different “experienced ”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “Yin et al. (2019) measure the frequency with which they revise their predictions to match the predictions of the model and their self-reported levels of trust in the model.”