A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy

By Emma Beede, E. Baylor, Fred Hersch, A. Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, Laura M. Vardoulakis. 2020

609 citations recorded by the source corpus.

Deep learning algorithms promise to improve clinician workflows and patient outcomes. However, these gains have yet to be fully demonstrated in real world clinical settings. In this paper, we describe a human-centered study of a deep learning system used in clinics for the detection of diabetic eye disease. From interviews and observation across eleven clinics in Thailand, we characterize current eye-screening workflows, user expectations for an AI-assisted screening process, and post-deployment experiences. Our findings indicate that several socio-environmental factors impact model performance, nursing workflows, and the patient experience. We draw on these findings to reflect on the value of conducting human-centered evaluative research alongside prospective evaluations of model accuracy.

Subfields: Computer Vision and Pattern Recognition, Ophthalmology, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Ophthalmology

Eric J. Topol

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “While current strategies exist to mitigate these concerns, addressing underspecification, and generalization issues more generally, could reduce a number of points of friction at the point of care (Beede et al., 2020).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “It says nothing of how the AI will perform in the chaos of a real-world environment” [3].”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “Ambient variations in this pipeline may result in unexpected performance degradation, such as degradations experienced by health care providers in Thailand deploying laboratory-tuned diabetic retinopathy classifiers in the field [2].”
Toward Causal Representation Learning (2021, background): “However, if we train a clinical system on doctors’ actions in controlled settings, the system will likely provide little additional insight compared to the doctors’ knowledge and may fail in surprising ways when deployed [19].”
Towards Causal Representation Learning (2021, background): “will likely provide little additional insight compared to the doctors’ knowledge and may fail in surprising ways when deployed [18].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “…different hospitals and deployment sites has also been shown to degrade model accuracy in other medical applications such as diabetic retinopathy (Beede et al., 2020) and chest radiographs (Zech et al., 2018; Phillips et al., 2020), including recent work on COVID-19 detection (DeGrave et al.,…”
Deep learning-enabled medical computer vision (2021, background): “For instance, regions of high-throughput healthcare, such as India and Thailand, have welcomed the deployment of technologies such as diabetic retinopathy screening systems 38 .”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “’s evaluation of real-world deep learning systems [17] point to the”
Harnessing multimodal data integration to advance precision oncology (2021, background): “Beyond centre-specific confounders, the clinical environment has unpredictable effects on model performance, often leading to substantial performance decreases 133 .”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “AI assistance improves human performance compared to without any assistance; however, the performance of human-AI teams seldom surpasses AI alone in challenging prediction tasks [3, 5, 7, 20, 21, 32, 33, 37, 43, 54, 65, 66, 73].”
Second opinion needed: communicating uncertainty in medical machine learning (2021, background): “As these advancements translate into real-world clinical decision tools, many are taking stock of what capabilities these systems presently lack 4 , especially in light of some mixed results from prospective validation efforts 3,5,6 .”
Do as AI say: susceptibility in deployment of clinical decision-aids (2021, methodology): “Weselected eight cases, each with a chest X-ray, from the open source MIMIC Chest X-ray database 20 .”
A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis (2019, background): “The study of interaction between users and models is of growing importance and is having a signiﬁcant impact on the e ﬃ cacy of Deep Active Learning systems and their deployment to real-world applications, especially in clinical scenarios (Beede et al. (2020); Amrehn et al. (2019)).”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, neutral): “theory [91, 103] and affinity diagramming [14, 64, 148, 149].”
Deep learning in histopathology: the path to the clinic (2021, background): “Such validation gives an indication of how well the algorithm performs in a new diagnostic situation, and can uncover problems with generalizability 128,129 .”
AI in health and medicine (2022, methodology): “Deep learning models have been applied widely in the area of ophthalmology, making important advances toward deployment 7,37–41 .”