A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.

By Xiaoxuan Liu, L. Faes, A. Kale, S. Wagner, D. J. Fu, A. Bruynseels, T. Mahendiran, Gabriella Moraes, Mohith Shamdas, C. Kern and 7 more. The Lancet Digital Health, 2019

1,444 citations recorded by the source corpus.

BACKGROUND Deep learning offers considerable promise for medical diagnostics. We aimed to evaluate the diagnostic accuracy of deep learning algorithms versus health-care professionals in classifying diseases using medical imaging. METHODS In this systematic review and meta-analysis, we searched Ovid-MEDLINE, Embase, Science Citation Index, and Conference Proceedings Citation Index for studies published from Jan 1, 2012, to June 6, 2019. Studies comparing the diagnostic performance of deep learning models and health-care professionals based on medical imaging, for any disease, were included. We excluded studies that used medical waveform data graphics material or investigated the accuracy of image segmentation rather than disease classification. We extracted binary diagnostic accuracy data and constructed contingency tables to derive the outcomes of interest: sensitivity and specificity. Studies undertaking an out-of-sample external validation were included in a meta-analysis, using a unified hierarchical model. This study is registered with PROSPERO, CRD42018091176. FINDINGS Our search identified 31 587 studies, of which 82 (describing 147 patient cohorts) were included. 69 studies provided enough data to construct contingency tables, enabling calculation of test accuracy, with sensitivity ranging from 9·7% to 100·0% (mean 79·1%, SD 0·2) and specificity ranging from 38·9% to 100·0% (mean 88·3%, SD 0·1). An out-of-sample external validation was done in 25 studies, of which 14 made the comparison between deep learning models and health-care professionals in the same sample. Comparison of the performance between health-care professionals in these 14 studies, when restricting the analysis to the contingency table for each study reporting the highest accuracy, found a pooled sensitivity of 87·0% (95% CI 83·0-90·2) for deep learning models and 86·4% (79·9-91·0) for health-care professionals, and a pooled specificity of 92·5% (95% CI 85·1-96·4) for deep learning models and 90·5% (80·6-95·7) for health-care professionals. INTERPRETATION Our review found the diagnostic performance of deep learning models to be equivalent to that of health-care professionals. However, a major finding of the review is that few studies presented externally validated results or compared the performance of deep learning models and health-care professionals using the same sample. Additionally, poor reporting is prevalent in deep learning studies, which limits reliable interpretation of the reported diagnostic accuracy. New reporting standards that address specific challenges of deep learning could improve future studies, enabling greater confidence in the results of future evaluations of this promising technology. FUNDING None.

Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/34f360a2bc49027bba77385be6c02179ba6875d8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

AI in health and medicine (2022, background): “Although the majority of studies have focused on a head-to-head comparison of AI with humans 82 , real-life medical practice is more likely to involve human-in-the-loop setups, where in humans actively collaborate with AI systems and provide oversight 83,84 .”
Artificial Intelligence in Medicine: Today and Tomorrow (2020, neutral): “A long-awaited meta-analysis compared performances of deep learning software and radiologists in the ﬁeld of imaging-based diagnosis (45): although deep learning seems to be as eﬃcient as radiologist for diagnosis, the authors pointed that 99% of studies were found not to have a reliable design;…”
Artificial intelligence and machine learning for medical imaging: A technology review. (2021, neutral): “For instance, the diagnostic performance of DL models has demonstrated to be equivalent to that of health-care professionals for certain applications [23], such as skin-cancer detection [24] or breast cancer detection [25].”
Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review (2020, background): “Recent studies and reviews have been primarily focusing on the performance and influence of AI systems at a diagnostic level, such as for disease identification [37-42], and the influence of AI robotics in surgery and disease management [43-46]; however, there is a lack of studies reviewing and repo”
Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review (2021, methodology): “35 We only screened one 10th of these articles; therefore, our results are presented using confidence intervals to extrapolate them to the whole sample.”
The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcare. (2021, neutral): “44 The first systematic review was published of deep learning performance in detecting diseases from medical imaging 45 —this showed that deep learning models perform similarly to human professionals, and few studies externally validated their results or compared performance between humans and…”
MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification (2021, neutral): “32 Chest CT MC (3) 1,370 1,027 / 103 / 240 VesselMNIST3D Yang et al.”
Characteristics of publicly available skin cancer image datasets: a systematic review. (2021, background): “3,64 These findings highlight the dangers of implementing algorithms for widespread use on broad populations without dataset transparency, especially if algorithm training was undertaken using a restricted demographic cohort.”
Pitfalls in Machine Learning Research: Reexamining the Development Cycle (2020, neutral): “For example, Liu et al. [30] found that out of tens of thousands of papers on deep learning published in the medical imaging literature, only a tiny fraction were methodologically sound.”