Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets
By Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, Ghassan Hamarneh. IEEE journal of biomedical and health informatics, 2019
517 citations recorded by the source corpus.
We propose a multitask deep convolutional neural network, trained on multimodal data (clinical and dermoscopic images, and patient metadata), to classify the 7-point melanoma checklist criteria and perform skin lesion diagnosis. Our neural network is trained using several multitask loss functions, where each loss considers different combinations of the input modalities, which allows our model to be robust to missing data at inference time. Our final model classifies the 7-point checklist and skin condition diagnosis, produces multimodal feature vectors suitable for image retrieval, and localizes clinically discriminant regions. We benchmark our approach using 1011 lesion cases, and report comprehensive results over all 7-point criteria and diagnosis. We also make our dataset (images and metadata) publicly available online at http://derm.cs.sfu.ca.
Subfields: Artificial Intelligence, Epidemiology, Oncology
View Paper: https://www.semanticscholar.org/paper/a0b7b10d504929bdb8765b72ce67077fe82b5971
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Erica Schmidt
- James Y. Zou
- Anuj Pareek
- Maggie Haitian Wang
- Imon Banerjee
- Rachel Han
- Mert Yuksekgonul
- Aerin Kim
In Epidemiology
Citations in the corpus
- Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020, methodology): “Kawahara et al.(39) also used CNNs as feature extractors for imaging modalities but experimented with a unique multimodal multi-task loss function that considers multiple combinations of the input modalities.”
- A Survey of Deep Learning for Scientific Discovery (2020, background): “…learn useful hidden representations of the image data in general, so after training with self-supervision, this neural network can then be successfully and efficiently finetuned on a downstream task. simultaneously trained on data from different modalities, such as images and text [139, 238, 102].”
- Federated learning for medical image analysis: A survey (2023, unknown): “(2021) [123] Skin lesion classification HAM10000 [122] ISIC19 [124] Derm7pt [125] PAD-UFES [126] Dermoscopy CNN (EfficientNet)”
- Post-hoc Concept Bottleneck Models (2022, methodology): “Following the setting in (Lucieri et al., 2020), we collect concepts from the Derm7pt (Kawahara et al., 2018) dataset.”
- Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset (2021, background): “…of PAD-UFES-20 [42], none of the publicly available data sets identified by the Sixth ISIC Skin Image Analysis Work-shop at CVPR 2021 (Derm7pt [30], Dermofit Image Li-brary, ISIC 2018 [16, 52], ISIC 2019 [15, 52, 18], ISIC 2020[46, 17], MED-NODE [27], PH2 [37], SD-128 [49], SD-198, SD-260)…”