Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes
By Qian Yang, Aaron Steinfeld, John Zimmerman. 2019
303 citations recorded by the source corpus.
Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of Unremarkable Computing, that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.
Subfields: Applied Psychology, Health Informatics, Health Information Management
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Psychology
In Health Informatics
In Health Information Management
Citations in the corpus
- A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy (2020, background): “Furthermore, as Yang and colleagues note, when HCI researchers attempt to study AI systems in a hospital or clinic, they are often prevented from fully embedding into clinical workflows and from evaluating systems using authentic patient data [42].”
- "Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “Despite the high-performing nature of these algorithms, prior work has shown that CDSSes can be difficult to successfully integrate into practice, citing a lack of HCI consideration as one of the primary reasons for failure [41, 44, 55, 76, 78].”
- Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design (2020, background): “First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
- Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Otherwise one may risk deploying an AI system un-integrated into individual and organizational workflows [66, 106], potentially resulting in misuse, mistrust [108, 109], or having profound ethical risks and unintended consequences, especially for marginalized groups [72, 86, 99].”
- Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “In addition, there have been interactive systems that provide AI assistance for complicated tasks beyond constrained prediction tasks [6, 70, 71].”