How much data is needed to train a medical image deep learning system to achieve necessary high accuracy

By Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, Synho Do. 2015

385 citations recorded by the source corpus.

The use of Convolutional Neural Networks (CNN) in natural image classification systems has produced very impressive results. Combined with the inherent nature of medical images that make them ideal for deep-learning, further application of such systems to medical image classification holds much promise. However, the usefulness and potential impact of such a system can be completely negated if it does not reach a target accuracy. In this paper, we present a study on determining the optimum size of the training data set necessary to achieve high classification accuracy with low variance in medical image classification systems. The CNN was applied to classify axial Computed Tomography (CT) images into six anatomical classes. We trained the CNN using six different sizes of training data set (5, 10, 20, 50, 100, and 200) and then tested the resulting system with a total of 6000 CT images. All images were acquired from the Massachusetts General Hospital (MGH) Picture Archiving and Communication System (PACS). Using this data, we employ the learning curve approach to predict classification accuracy at a given training sample size. Our research will present a general methodology for determining the training data set size necessary to achieve a certain target classification accuracy that can be easily applied to other problems within such systems.

Subfields: Artificial Intelligence, Biomedical Engineering, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Medical Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

In Medical Imaging

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “A number of prior studies observed power-law relations between the generalization error and training data size (Cho et al., 2015; Miceli Barone et al., 2017; Johnson et al., 2018).”
Opening the Black Box of Deep Neural Networks via Information (2017, background): “Another fundamental issue in machine learning, which we only deal with briefly in this paper, is the dependence on the training data size [Cho et al. (2015)].”
Arti ﬁ cial Intelligence and Machine Learning in Radiology: Opportunities, Challenges, Pitfalls, and Criteria for Success (background): “Failure to assemble a suf ﬁ ciently large enough training set is a potential pitfall that could have the effect of making the results less accurate or generalizable [19].”
Deep learning in medical imaging and radiation therapy. (2018, background): “16,63,193,302,303 Mohamed et al. 193 found that for breast density classification, there is a small increase in test performance (the area under the receiver operating characteristic curve increases from 0.95 to 0.99, P < 0.001) when their training set size increased from 2000 images to 6000 images.”
Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions (2017, background): “While some authors report that their supervised frameworks require only one training sample [28], most researchers report that their results were consistently improving with an increase in size of training datasets [58, 59].”
A scoping review of transfer learning research on medical image analysis using ImageNet (2020, result): “Congruent with previous findings [140], this suggests that”
Is your dataset big enough? Sample size requirements when using artificial neural networks for discrete choice analysis (2018, background): “One rule-of-thumb is that the sample size needs to be at least a factor 50 to 1000 times the number of prediction classes (which, in the choice modelling context, is the choice set size) (Cho et al., 2015; Cire ş an et al., 2012).”
Predicting accuracy on large datasets from smaller pilot data (2018, methodology): “The power-law models used in this paper have been investigated many times in prior literature (Haussler et al., 1996; Mukherjee et al., 2003; Figueroa et al., 2012; Beleites et al., 2013; Hajian-Tilaki, 2014; Cho et al., 2015).”
Revisiting Neural Scaling Laws in Language and Vision (2022, methodology): “While theoretical arguments alone seldom predict scaling law parameters in modern neural architectures [2, 21, 32], it has been observed that the benefit of scale could be predicted empirically [3, 4, 9, 17, 18, 20, 22, 23, 28, 30, 31].”