Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists

By H. Haenssle, Christian Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, Aadi Kalloo, A. Hassen, L. Thomas, Alexander Enk and 52 more. Annals of Oncology, 2018

1,405 citations recorded by the source corpus.

Background Deep learning convolutional neural networks (CNN) may facilitate melanoma detection, but data comparing a CNN's diagnostic performance to larger groups of dermatologists are lacking. Methods Google's Inception v4 CNN architecture was trained and validated using dermoscopic images and corresponding diagnoses. In a comparative cross-sectional reader study a 100-image test-set was used (level-I: dermoscopy only; level-II: dermoscopy plus clinical information and images). Main outcome measures were sensitivity, specificity and area under the curve (AUC) of receiver operating characteristics (ROC) for diagnostic classification (dichotomous) of lesions by the CNN versus an international group of 58 dermatologists during level-I or -II of the reader study. Secondary end points included the dermatologists' diagnostic performance in their management decisions and differences in the diagnostic performance of dermatologists during level-I and -II of the reader study. Additionally, the CNN's performance was compared with the top-five algorithms of the 2016 International Symposium on Biomedical Imaging (ISBI) challenge. Results In level-I dermatologists achieved a mean (±standard deviation) sensitivity and specificity for lesion classification of 86.6% (±9.3%) and 71.3% (±11.2%), respectively. More clinical information (level-II) improved the sensitivity to 88.9% (±9.6%, P = 0.19) and specificity to 75.7% (±11.7%, P < 0.05). The CNN ROC curve revealed a higher specificity of 82.5% when compared with dermatologists in level-I (71.3%, P < 0.01) and level-II (75.7%, P < 0.01) at their sensitivities of 86.6% and 88.9%, respectively. The CNN ROC AUC was greater than the mean ROC area of dermatologists (0.86 versus 0.79, P < 0.01). The CNN scored results close to the top three algorithms of the ISBI 2016 challenge. Conclusions For the first time we compared a CNN's diagnostic performance with a large international group of 58 dermatologists, including 30 experts. Most dermatologists were outperformed by the CNN. Irrespective of any physicians' experience, they may benefit from assistance by a CNN's image classification. Clinical trial number This study was registered at the German Clinical Trial Register (DRKS-Study-ID: DRKS00013570; https://www.drks.de/drks_web/).

Subfields: Artificial Intelligence, Health Informatics, Oncology

View Paper: https://www.semanticscholar.org/paper/546785490ac417be1f83ced6a8272e934934f411

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Oncology

Citations in the corpus

A guide to deep learning in healthcare (2019, background): “…deep-learning models have achieved physician-level accuracy at a broad variety of diagnostic tasks, including identifying moles from melanomas 9,10 , diabetic retinopathy, cardiovascular risk, and referrals from fundus 15,16 and optical coherence tomography (OCT) 17 images of the eye, breast…”
Revolutionizing healthcare: the role of artificial intelligence in clinical practice (2023, background): “Furthermore, a study utilized deep learning to detect skin cancer which showed that an AI using CNN accurately diagnosed melanoma cases compared to derma-tologists and recommended treatment options [13, 14].”
The Development of a Skin Cancer Classification System for Pigmented Skin Lesions Using Deep Learning (2020, result): “Many similar tests have been reported in previous research [3,36,37], and it is considered that the machine learning algorithm has reached dermatologist-level accuracy in skin lesion classiﬁcation [4,5,36].”
Attention-Based Prototypical Learning Towards Interpretable, Confident and Robust Deep Neural Networks (2019, positive): “By refraining from some predictions, we demonstrate unprecedentedly high AUC values without using transfer learning or highly-customized models (Haenssle et al., 2018).”
Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study (2019, background): “Furthermore, Haenssle and colleagues 15 reported expert-level accuracy of algorithms for dermatoscopic images of melanocytic lesions.”
Artificial intelligence in medical education (2019, background): “Already, computers can perform prediction and diagnosis better than humans (Bogu (cid:1) sevi (cid:1) cius et al. 2002; Esteva et al. 2017; Litjens et al. 2017; Mobadersany et al. 2017; Betancur et al. 2018; De Fauw et al. 2018; Haenssle et al. 2018).”