Results of the 2016 International Skin Imaging Collaboration ISBI Challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images

By M. Marchetti, Noel C. F. Codella, S. Dusza, David Gutman, Brian Helba, Aadi Kalloo, N. Mishra, C. Carrera, Emre M. Celebi, J. Defazio and 6 more. Journal of American Academy of Dermatology, 2017

294 citations recorded by the source corpus.

Background Computer vision may aid in melanoma detection. Objective We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images. Methods We conducted a cross‐sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into “fusion” algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant. Results The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P = .68) but lower than the best‐performing fusion algorithm (59% vs. 76%, P = .02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P = .001). Limitations The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice. Conclusion Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Oncology

View Paper: https://www.semanticscholar.org/paper/7a28816ccfa2c7608ccba437d38d1e69de1f428e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Oncology

Citations in the corpus

Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study (2019, result): “Unlike in similar studies, 15,22,23 our test set included not only melanoma and nevi, but also non-melanocytic lesions.”
Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC) (2016, methodology): “4) The most complex fusion approach (NL-SVM) led to a decrease in performance, whereas simpler methods led to overall improvements in performance, consistent with previous findings [17].”
Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC) (2019, background): “Evaluation criteria historically had been the Jaccard index [3,4,5], averaged over all images in the dataset.”
Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review (2018, background): “The 25 participating teams did not apply traditional standard machine learning methods; instead, they all employed a deep learning technique: convolutional neural networks (CNNs) [13].”
Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists (2018, methodology): “In contrast to Marchetti et al. [15] other authors have not used ‘benchmark’ image datasets, and only a few studies included a small number of readers for comparison with their designed computer algorithms [13, 20].”
Chatbot for Health Care and Oncology Applications Using Artificial Intelligence and Machine Learning: Systematic Review (2021, unknown): “Studies have shown that the interpretation of medical images for the diagnosis of tumors performs equally well or better with AI compared with experts [53-56].”
Collaborative Human-AI (CHAI): Evidence-Based Interpretable Melanoma Classification in Dermoscopic Images (2018, unknown): “From this repository, snapshots of the dataset have been used to host two consecutive years of benchmark challenges [1,2], which have increased interest in the computer vision community [2,3,4,5,6], and supported the development of methods that surpassed the diagnostic performance of expert clinicia”