Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making

By Carrie J. Cai, Emily Reif, Narayan Hegde, J. Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda B. Viégas, Greg S. Corrado, Martin C. Stumpe and 1 more. 2019

468 citations recorded by the source corpus.

Machine learning (ML) is increasingly being used in image retrieval systems for medical decision making. One application of ML is to retrieve visually similar medical images from past patients (e.g. tissue from biopsies) to reference when making a medical decision with a new patient. However, no algorithm can perfectly capture an expert's ideal notion of similarity for every case: an image that is algorithmically determined to be similar may not be medically relevant to a doctor's specific diagnostic needs. In this paper, we identified the needs of pathologists when searching for similar images retrieved using a deep learning algorithm, and developed tools that empower users to cope with the search algorithm on-the-fly, communicating what types of similarity are most important at different moments in time. In two evaluations with pathologists, we found that these tools increased the diagnostic utility of images found and increased user trust in the algorithm. The tools were preferred over a traditional interface, without a loss in diagnostic accuracy. We also observed that users adopted new strategies when using refinement tools, re-purposing them to test and understand the underlying algorithm and to disambiguate ML errors from their own errors. Taken together, these findings inform future human-ML collaborative systems for expert decision-making.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “[17] investigated how a medical AI can assist with doctors’ decision-making process during prostate cancer diagnosis, by helping them compare and contrast similar images.”
A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy (2020, background): “Logistically, researchers have faced hurdles conducting studies within clinical environments [42], and often are limited to lab-based studies where clinicians use retrospective patient data [8, 9], or synthetic patient data in order to complete simulated tasks.”
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use (2019, background): “to know the samples the model has previously seen Cai et al. (2019). However, in time constrained settings such as the ICU or ED, clinicians did not find this explanation technique appealing.”
The Metacognitive Demands and Opportunities of Generative AI (2023, background): “Target areas for this include the explain-ability and customizability of GenAI systems.”
Identifying Challenges and Opportunities in Human-AI Collaboration in Healthcare (2019, background): “[3, 12] • How does the potential effect of AI technology on the agency and power of healthcare workers within organizational contexts affect trust in an AI system?”
Similar image search for histopathology: SMILY (2019, methodology): “To address the lack of algorithmic awareness for the search intent, advances in human-computer interaction may enable interactive refinement of search results based on certain desired axes of similarity.(24) The potential use cases for a tool like SMILY can be categorized into diagnosis, research, a”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “Also called Explanation by Example , What-else explanations pick samples from the model’s training dataset that are similar to the original input in the model representation space [30].”
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making (2019, background): “Prostate cancer is one of the most common types of cancer, and the subject of significant attention in recent work on machine learning-powered cancer diagnosis [11, 15, 35, 56, 57, 69].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Acquisition of chess knowledge in AlphaZero (2021, background): “For concept-based explanation, the alignment between the representation and a human’s mental model of the concept remains an open problem [12, 41, 42].”