"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

By Carrie J. Cai, Samantha Winter, David F. Steiner, Lauren Wilcox, Michael Terry. 2019

558 citations recorded by the source corpus.

Although rapid advances in machine learning have made it increasingly applicable to expert decision-making, the delivery of accurate algorithmic predictions alone is insufficient for effective human-AI collaboration. In this work, we investigate the key types of information medical experts desire when they are first introduced to a diagnostic AI assistant. In a qualitative lab study, we interviewed 21 pathologists before, during, and after being presented deep neural network (DNN) predictions for prostate cancer diagnosis, to learn the types of information that they desired about the AI assistant. Our findings reveal that, far beyond understanding the local, case-specific reasoning behind any model decision, clinicians desired upfront information about basic, global properties of the model, such as its known strengths and limitations, its subjective point-of-view, and its overall design objective--what it's designed to be optimized for. Participants compared these information needs to the collaborative mental models they develop of their medical colleagues when seeking a second opinion: the medical perspectives and standards that those colleagues embody, and the compatibility of those perspectives with their own diagnostic patterns. These findings broaden and enrich discussions surrounding AI transparency for collaborative decision-making, providing a richer understanding of what experts find important in their introduction to AI assistants before integrating them into routine practice.

Subfields: Artificial Intelligence, Health Informatics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

Citations in the corpus

A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy (2020, background): “In subsequent work, Cai et al. examined what information pathologists (in a lab setting) found to be important when being introduced to AI assistants, before integrating these assistants into routine prostate cancer screening practice [9].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “For example, some studies asked open-ended “why” questions following survey scales, others conducted exit interviews or asked participants to think aloud while using the AI system [15, 19, 22, 25, 30, 48, 61, 69, 124, 125, 140].”
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions (2022, methodology): “Such a dataset would be useful, as model explanations can aid humans in tasks such as medical diagnosis (Cai et al., 2019; Lundberg et al., 2018), data annotation (Schmidt and Biessmann, 2019) and deception detection (Lai and Tan, 2019).”
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions (2022, background): “Previous studies have reported that model-generated explanations can aid humans in some tasks (Cai et al., 2019; Lundberg et al., 2018; Schmidt and Biessmann, 2019; Lai and Tan, 2019), though only when the models are generally able to outperform humans at that task (Bansal et al., 2021).”
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models (2020, background): “Previous research suggests beneﬁts of educating users about the AI and its capabilities [1], or providing onboarding materials and exercises [6].”
How Machine Learning will Transform Biomedicine (2020, unknown): “‘Transparency’ in this context includes description of the optimized objectives, strengths, quantitative performance and limitations of a particular algorithm (Cai et al., 2019) as well as the procedures used to validate the algorithm.”
How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India (2022, background): “On-boarding users to AI systems via guides may be a viable first step to align users’ mental models with AI systems [29].”
Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021, methodology): “In terms of post-hoc explainability, perturbation-based models that use model-agnostic explanations are common in the literature [98,99,118,120,127] and have been used in two of the proposed CDSS [118,120].”
Towards accurate differential diagnosis with large language models (2023, unknown): “trusted the outputs of the model or their understanding of its training and limitations, or undertake focused ‘onboarding’ or training in its use, which are all known to be important modifiers of the benefits derived by clinicians from AI assistants 32 .”
Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021, background): “In interviews, clinicians discussed their information desiderata when onboarding new AI decision support: strengths, weaknesses, point-of-views [6].”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “users of an AI system supporting medical diagnosis [18, 63, 104] , they may want to see a high-level global explanation about the model during the onboarding stage to form an appropriate mental model, but have higher demand for local explanations, especially when they get unexpected or suspicious pr”
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023, unknown): “Clear communication and user education about the capabilities and limitations of the system will be essential (Cai et al., 2019).”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, methodology): “…together from the start, with HCI methodologies applied at all stages of interpretability tool development: supporting need-ﬁnding studies (e.g., [8, 64]), designing tools that can be understood by users with different background (e.g., [37]), and undertaking user studies at each stage of tool…”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, background): “[10] described the onboarding phase to an XAI system as a key phase that forms users’ initial impressions of an XAI system.”
Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems (2022, background): “The term ’human-AI collaboration’ has emerged in recent work studying user interaction with AI systems [7, 24, 118, 151].”
Do as AI say: susceptibility in deployment of clinical decision-aids (2021, methodology): “This may take the form of a human-AI calibration phase, where a library of curated test cases is used to demonstrate the system ’ s strengths and weaknesses, or an AI primer during on-boarding 40 .”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “Indeed, at first glance, it appears that researchers have already confirmed the utility of explanations on tasks ranging from medical diagnosis [10, 47] and data annotation [57] to deception detection [36].”