Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
By Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, Jianming Liang. 2016
2,955 citations recorded by the source corpus.
Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Medical Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Kristen Grauman
- Li Fei-Fei
- Trevor Darrell
- Rogério Schmidt Feris
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
In Radiology, Nuclear Medicine and Imaging
In Medical Imaging
- Xiaosong Wang
- Ronald M. Summers
- Mohammad Tariqul Islam
- Maciej A. Mazurowski
- Mustafa R. Bashir
- Mateusz Buda
- Ashirbani Saha
- Muhammad Abdul Kadir
Citations in the corpus
- Quantifying the Carbon Emissions of Machine Learning (2019, background): “Notably, recent research has shown that using pre-trained models with task-specific fine-tuning performs as well as training from scratch, while being more robust, for tasks in image recognition [7, 8] and NLP [9].”
- Deep Learning for Health Informatics (2017, background): “fine tuning have been suggested in medical imaging applications [58], [72], [74], [76].”
- SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, background): “Alternatively, the last few layers of the deep network can be fine-tuned while freezing the parameters of the remaining initial layers to their pre-trained values [44, 1].”
- MRI-Based Brain Tumor Classification Using Ensemble of Deep Features and Machine Learning Classifiers (2021, background): “This reduces the long training time that is normally required for training deep learning models from scratch and also removes the requirement of having a large dataset for the training model [48,49].”
- Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images (2021, positive): “Thus, it opens the opportunity of utilizing smaller datasets and also reduces the time required to develop a deep learning algorithm from scratch [24,25].”
- Spatial aggregation of holistically‐nested convolutional neural networks for automated pancreas localization and segmentation☆ (2017, background): “27% is achieved, at the computational cost of 2–3 min, not hours as in Wang et al. (2014c) , Chu et al.”
- A Systematic Review on Model Watermarking for Neural Networks (2020, background): “Thereby, it is possible to improve a model for certain kinds of data (Sharif Razavian et al., 2014; Tajbakhsh et al., 2016).”
- Deep Bayesian Hashing with Center Prior for Multi-modal Neuroimage Retrieval (2020, background): “NEUROIMAGE analysis has made a profound contribution to modern clinical analysis [1]–[3], image-guided surgery [4]–[6], and automated diagnostic studies [7]–[13].”
- Machine Learning in Radiation Oncology: Opportunities, Requirements, and Needs (2018, methodology): “In applications in which deep learning has been successfully applied, the models have been trained with tens of thousands observations (4, 36).”
- Low Latency Privacy Preserving Inference (2018, background): “Hence, it is a common practice to use deep representations and train only the top layer(s) (Yosinski et al., 2014; Tajbakhsh et al., 2016).”
- Integrating Physics-Based Modeling with Machine Learning: A Survey (2020, methodology): “This is similar to the common application of pre-training in computer vision, where CNNs are often pre-trained with very large image datasets before being fine-tuned on images from the task at hand [243].”
- VPE: Variational Policy Embedding for Transfer Reinforcement Learning (2018, background): “To address these issues, it has been suggested to learn a single policy that generalizes well to similar environments [7]–[9], fine-tune policies in new environments [10]–[13], or learn from teacher policies [14]–[17].”
- DeepSigns: An End-to-End Watermarking Framework for Ownership Protection of Deep Neural Networks (2019, background): “These attacks include parameter pruning (e.g. [27, 28]), model fine-tuning (e.g. [29–32]) and watermark overwriting (e.g. [11, 33]).”