Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset
By Matthew Groh, Caleb Harris, Luis R. Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, Omar Badri. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021
306 citations recorded by the source corpus.
How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is under-represented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.
View Paper: https://www.semanticscholar.org/paper/069b9321f85b6bc005fff3c254d34c791f6c0741
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Weixin Liang
- James Y. Zou
- R. Daneshjou
- Kailas Vodrahalli
- R. Novoa
- Melissa Jenkins
- Mert Yuksekgonul
- V. Rotemberg
Citations in the corpus
- Disparities in dermatology AI performance on a diverse, curated clinical image set (2021, background): “Images from online atlases such as Fitzpatrick 17k do not have histopathological confirmation of malignancies and likely have label noise—a subset of 504 images were reviewed by board-certified dermatologists, and only 69% of the images appeared diagnostic of the labeled condition (10).”
- Post-hoc Concept Bottleneck Models (2022, methodology): “We further add the dark-skin-color concept to test for biases in these models, where we obtain the concept bank images from the Fitzpatrick17k(Groh et al., 2021) dataset.”