Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition.

By J. Winkler, C. Fink, F. Toberer, Alexander Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz and 1 more. 2019

352 citations recorded by the source corpus.

Importance Deep learning convolutional neural networks (CNNs) have shown a performance at the level of dermatologists in the diagnosis of melanoma. Accordingly, further exploring the potential limitations of CNN technology before broadly applying it is of special interest. Objective To investigate the association between gentian violet surgical skin markings in dermoscopic images and the diagnostic performance of a CNN approved for use as a medical device in the European market. Design and Setting A cross-sectional analysis was conducted from August 1, 2018, to November 30, 2018, using a CNN architecture trained with more than 120 000 dermoscopic images of skin neoplasms and corresponding diagnoses. The association of gentian violet skin markings in dermoscopic images with the performance of the CNN was investigated in 3 image sets of 130 melanocytic lesions each (107 benign nevi, 23 melanomas). Exposures The same lesions were sequentially imaged with and without the application of a gentian violet surgical skin marker and then evaluated by the CNN for their probability of being a melanoma. In addition, the markings were removed by manually cropping the dermoscopic images to focus on the melanocytic lesion. Main Outcomes and Measures Sensitivity, specificity, and area under the curve (AUC) of the receiver operating characteristic (ROC) curve for the CNN's diagnostic classification in unmarked, marked, and cropped images. Results In all, 130 melanocytic lesions (107 benign nevi and 23 melanomas) were imaged. In unmarked lesions, the CNN achieved a sensitivity of 95.7% (95% CI, 79%-99.2%) and a specificity of 84.1% (95% CI, 76.0%-89.8%). The ROC AUC was 0.969. In marked lesions, an increase in melanoma probability scores was observed that resulted in a sensitivity of 100% (95% CI, 85.7%-100%) and a significantly reduced specificity of 45.8% (95% CI, 36.7%-55.2%, P < .001). The ROC AUC was 0.922. Cropping images led to the highest sensitivity of 100% (95% CI, 85.7%-100%), specificity of 97.2% (95% CI, 92.1%-99.0%), and ROC AUC of 0.993. Heat maps created by vanilla gradient descent backpropagation indicated that the blue markings were associated with the increased false-positive rate. Conclusions and Relevance This study's findings suggest that skin markings significantly interfered with the CNN's correct diagnosis of nevi by increasing the melanoma probability scores and consequently the false-positive rate. A predominance of skin markings in melanoma training images may have induced the CNN's association of markings with a melanoma diagnosis. Accordingly, these findings suggest that skin markings should be avoided in dermoscopic images intended for analysis by a CNN. Trial Registration German Clinical Trial Register (DRKS) Identifier: DRKS00013570.

Subfields: Artificial Intelligence, Health Informatics, Oncology, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Oncology

Cynthia Rudin

In Medical Imaging

Cynthia Rudin

Citations in the corpus

A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, methodology): “If this reasoning process were correct, it would lead to a higher chance that: (1) the model could generalize beyond its finite training set; (2) the model’s reasoning process could be useful information for doctors, even if its prediction is sometimes incorrect; (3) it would be much easier to troub”
Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations (2020, background): “In some cases, this has led to unexpected and/or unwanted behavioral artifacts of trained networks (Yudkowsky, 2008; Ribeiro et al., 2016; Winkler et al., 2019).”
Key challenges for delivering clinical impact with artificial intelligence (2019, background): “The presence of surgical skin markings have also been shown to falsely increase a deep learning model ’ s melanoma probability scores and hence false positive rate [73].”
A Survey of Deep Learning for Scientific Discovery (2020, background): “Deep learning models are notorious for relying on spurious correlations in the data to perform their predictions [8, 162, 247].”
Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset (2021, background): “The lack of consideration of subgroups within a population has been shown to lead deep neural networks to produce large accuracy disparities across gender and skin color for facial recognition [11], across images with and without surgical markings in dermatology [56, 10], and across treated and untr”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging (2019, background): “Non-causal confounding features such as healthcare process quantities can also contribute substantially to high model performance on data subsets heavily associated with these confounding variables [1, 2, 36, 41].”
Artiﬁcial Intelligence Applications in Dermatology: Where Do We Stand? (neutral): “For example, a study by Winkler et al. showed that surgical markings signiﬁcantly interfered with the ability of a system to correctly diagnose dermatoscopic images of melanoma and increased the false positive rate (89).”