Big Self-Supervised Models Advance Medical Image Classification

By Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zach Beaver, J. Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting-Li Chen and 2 more. 2021

741 citations recorded by the source corpus.

Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pre-training strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning on unlabeled domain-specific medical images significantly improves the accuracy of medical image classifiers. We introduce a novel Multi-Instance Contrastive Learning (MICLe) method that uses multiple images of the underlying pathology per patient case, when available, to construct more informative positive pairs for self-supervised learning. Combining our contributions, we achieve an improvement of 6.7% in top-1 accuracy and an improvement of 1.1% in mean AUC on dermatology and chest X-ray classification respectively, outperforming strong supervised baselines pretrained on ImageNet. In addition, we show that big self-supervised models are robust to distribution shift and can learn efficiently with a small number of labeled medical images.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “…by the recent line of work on image view-based contrastive learning (H´enaﬀ et al., 2020; Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Sowrirajan et al., 2021; Azizi et al., 2021), but fundamentally diﬀers from existing studies by exploiting contrastive learning using the text modality.”
ImageNet-21K Pretraining for the Masses (2021, background): “Many papers used these pretrained models for downstream tasks ( [55, 35, 31, 1] for example).”
Continual Self-Supervised Learning: Towards Universal Multi-Modal Medical Data Representation Learning (2023, background): “A review of existing medical SSL approaches highlights a pattern: they primarily focus on a single modality data, such as X-rays [2, 4, 6, 16, 39, 56, 57], computed tomography (CT) scans [16, 19, 41, 50, 52, 53, 56–58], magnetic resonance image (MRI) scans [6, 22, 33, 55], pathological images [23,…”
Self-supervised learning in medicine and healthcare (2022, background): “One study 14 worked with chest X-ray data with both lateral and frontal scans and with dermatology data with photos from different perspectives (where only a single view was available, a data-augmentation step was used to generate image pairs; where two views existed, a minor data-augmentation…”
AUC Maximization in the Era of Big Data and AI: A Survey (2022, neutral): “This approach was recently explored in [4, 151, 199].”
MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation (2021, methodology): “Azizi et al. (2021) also found improvement in dermatology classiﬁcation when applying a second contrastive pretraining stage where strictly distinct images from the same patient are selected as positive pairs.”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, background): “Furthermore, several recent papers that apply self-supervised learning to domain-specific problems did not apply transfer learning to the pretraining process itself, which motivated our work [2, 60, 1, 32].”
A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications (2023, background): “[70] to improve the learned features of DL models by training them on a large number of unlabelled images of a specific task then the models will be trained on a small, labeled dataset for that same task.”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “This idea is generalized to inpainting tasks [18, 37, 57] to learn visual representations [3,7,46] by predicting the original image patches.”