Predicting sample size required for classification performance

By Rosa L. Figueroa, Qing Zeng-Treitler, S. Kandula, Long H. Ngo. BMC Medical Informatics and Decision Making, 2012

512 citations recorded by the source corpus.

BackgroundSupervised learning methods need annotated data in order to generate efficient models. Annotated data, however, is a relatively scarce resource and can be expensive to obtain. For both passive and active learning methods, there is a need to estimate the size of the annotated sample required to reach a performance target.MethodsWe designed and implemented a method that fits an inverse power law model to points of a given learning curve created using a small annotated training set. Fitting is carried out using nonlinear weighted least squares optimization. The fitted model is then used to predict the classifier's performance and confidence interval for larger sample sizes. For evaluation, the nonlinear weighted curve fitting method was applied to a set of learning curves generated using clinical text and waveform classification tasks with active and passive sampling methods, and predictions were validated using standard goodness of fit measures. As control we used an un-weighted fitting method.ResultsA total of 568 models were fitted and the model predictions were compared with the observed performances. Depending on the data set and sampling method, it took between 80 to 560 annotated samples to achieve mean average and root mean squared error below 0.01. Results also show that our weighted fitting method outperformed the baseline un-weighted method (p < 0.05).ConclusionsThis paper describes a simple and effective sample size prediction algorithm that conducts weighted fitting of learning curves. The algorithm outperformed an un-weighted algorithm described in previous literature. It can help researchers determine annotation sample size for supervised machine learning.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/2f31cfb36a8d0c9b2039eee1b5848f618d1a50dc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Current Applications and Future Impact of Machine Learning in Radiology. (2018, unknown): “On the other hand, performance classification tasks (eg, classifying a liver lesion as malignant vs benign) may require many more label examples, which may also be largely dependent on the number of classifiers to distinguish between (79).”
How much data is needed to train a medical image deep learning system to achieve necessary high accuracy (2015, methodology): “Several approaches to this question have been introduced and explored in different applications (Figueroa et al. (2012), Beleites et al. (2013), Dobbin & Simon (2007), Dobbin et al. (2008)).”
Revisiting Neural Scaling Laws in Language and Vision (2022, methodology): “This is, perhaps, the most commonly used model in the literature; see for instance [1, 13, 17, 19, 20, 22, 28, 30, 31].”
Predicting accuracy on large datasets from smaller pilot data (2018, methodology): “The power-law models used in this paper have been investigated many times in prior literature (Haussler et al., 1996; Mukherjee et al., 2003; Figueroa et al., 2012; Beleites et al., 2013; Hajian-Tilaki, 2014; Cho et al., 2015).”
Sample size estimation in diagnostic test studies of biomedical informatics (2014, methodology): “Hanley and McNeil [7] used exponential approximation to estimate Q1 and Q2 as”
Supervised machine learning: A brief primer (2020, background): “Along these lines, the machine learning literature sometimes generates learning curves to plot how model performance would improve as the number of observations increases (Figueroa et al., 2012).”