Transfusion: Understanding Transfer Learning for Medical Imaging

By Maithra Raghu, Chiyuan Zhang, Jon M. Kleinberg, Samy Bengio. 2019

1,299 citations recorded by the source corpus.

Transfer learning from natural image datasets, particularly ImageNet, using standard large models and corresponding pretrained weights has become a de-facto method for deep learning applications to medical imaging. However, there are fundamental differences in data sizes, features and task specifications between natural image classification and the target medical tasks, and there is little understanding of the effects of transfer. In this paper, we explore properties of transfer learning for medical imaging. A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures. Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse. We isolate where useful feature reuse occurs, and outline the implications for more efficient model exploration. We also explore feature independent benefits of transfer arising from weight scalings.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “As a result, Raghu et al. (2019) found that ImageNet pretraining often provides little to no beneﬁt compared to simple random initialization.”
What is being transferred in transfer learning? (2020, methodology): “Directly related to our work is [36], where they investigate transfer learning from pre-trained ImageNetmodel to medical domain and note the role of model size on transfer performance, as well as the role of feature independent aspects such as weight scaling.”
Rethinking Pre-training and Self-training (2020, neutral): “[44] show that ImageNet pre-training does not”
Investigating Transferability in Pretrained Language Models (2020, methodology): “This methodology is distinct from the shuffled init methodology of Raghu et al. (2019) which involves permuting every parameter in their computer vision model, as opposed to permuting just the layers as we do.”
How Well Do Self-Supervised Models Transfer? (2020, background): “As we mentioned in the introduction, a major question we set out to answer is whether ImageNet performance is in general representative of downstream performance on diverse tasks and datasets? This determines whether practitioners can safely select the latest benchmark leading SSL methods for downst”
Parameter Prediction for Unseen Deep Architectures (2021, background): “We consider low-data tasks as they often benefit more from transfer learning [86, 87].”
Harnessing multimodal data integration to advance precision oncology (2021, background): “However, recent evidence shows that small models without pretraining, such as ResNet-50, for small medical imaging datasets can perform comparably to pretrained large models 63 .”
Accurate brain age prediction with lightweight deep neural networks (2019, background): “One insight is that in medical imaging (and neuroimaging) applications, the classification tasks involve only a few classes and thus there is no need for a wide FC layer or large over-parameterization (Raghu et al., 2019).”
A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises (2020, background): “However, ImageNet consists of natural images, and its pretrained models are for 2-D images only and are not necessarily the best for medical images, especially for small-sample settings [54].”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, background): “…et al., 2019) to revealing hidden linguistic structure in large language models (Bau et al., 2019; Kudugunta et al., 2019; Wu et al., 2019; 2020) to applications in neuroscience (Shi et al., 2019; Li et al., 2019; Merel et al., 2019; Zhang & Bellec, 2020) and medicine (Raghu et al., 2019).”
Big Self-Supervised Models Advance Medical Image Classification (2021, methodology): “Given the small size of the validation dataset and following [34, 37] suggestion, for the downstream task evaluations we randomly re-split the training set into 67,429 training images, 22,240 validation images, and 33,745 test images.”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, result): “We observe differences among objectives only in later network layers, but previous work has found that these layers change substantially during finetuning [79, 53, 46], an observation we replicate in Appendix D.”
A generalist vision–language foundation model for diverse biomedical tasks (2023, positive): “The practice of fine-tuning pretrained models, a widely acknowledged and highly effective approach in natural-language processing and computer vision, has also found important application in medical AI 76,77 .”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, neutral): “These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18], representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38].”