Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI

By Alon Jacovi, Ana Marasovic, Tim Miller, Yoav Goldberg. 2020

648 citations recorded by the source corpus.

Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, interpersonal trust (i.e., trust between people) as defined by sociologists. This model rests on two key properties: the vulnerability of the user; and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI model is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (that detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.

Subfields: Artificial Intelligence, Health Informatics, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2019) harm detectors to support better harm mitigation strategies (Lipton, 2018; Jacovi et al., 2021).”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Strategies of this type are one way to warrant trust in an ML system (Jacovi et al., 2020).”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “This trust may or may not be warranted (Jacovi et al., 2021); future work would be well-suited to further investigate generated explanations for incorrect label predictions such as in Appendix C, which could mislead end users. members of the Mosaic team at the Allen Institute for AI for valuable…”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “For the former, recent FATE literature has begun to unpack trust as a multi-faceted and multi-loci concept (Jacovi et al., 2021; Liao and Sundar, 2022).”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “Increasing the unwarranted trust can be highly problematic in high-stakes decisions [65].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “In this way, it could be possible to gain confidence that models will (or will not) behave in a safe way without necessarily conducting extensive testing of the models [Jacovi et al., 2021].”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, background): “Mitigating deployment bias might involve systems that help users balance their faith in model predictions with other information and judgements [28].”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, background): “Some researchers have explored various merits of explanations to humans, such as supporting human decision makings (Lai and Tan, 2019), increasing human trust in AI (Jacovi et al., 2020) and even teaching humans to perform challenging tasks (Lai et al.”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “Yet correlational predictive models can be untrustworthy (Jacovi et al., 2021): They may latch onto spurious correlations (“shortcuts”), leading to errors in out-of-distribution (OOD) settings (e.”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “Such contracts are AI functionalities that are anticipated to have known vulnerabilities 305 .”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “Interpreting supervised machine learning models by analyzing local explanations—justifications of their individual predictions1—is important for debugging, protecting sensitive information, understanding model robustness, and increasing warranted trust in a stated contract (Doshi-Velez and Kim, 2017”
Trust in AI: progress, challenges, and future directions (2024, positive): “In Jacovi et al. (2021b), a precise discussion is presented regarding the nature of trust in AI, as well as the prerequisites and goals of the cognitive mechanism of trust.”
A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective (2022, background): “A growing number of researchers argue that fostering and maintaining user trust is the key to calibrating the user-AI relationship (Jacovi et al., 2021; Shin, 2021), achieving trust-worthy AI (Smith, 2019), and further unlocking the potential of AI for society (Bughin et al., 2018; Cheatham et al.,…”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, background): “Philosophically, the honest student is motivated by the goal of building warranted trust in question anwering systems (Jacovi et al., 2021), through an architecture in which the rationales meaningfully constrain the predicted answer (DeYoung et al., 2020) and can easily be checked by users.”