Machine learning algorithm validation with a limited sample size
By Andrius Vabalas, Emma Gowen, Ellen Poliakoff, A. Casson. PLoS ONE, 2019
1,474 citations recorded by the source corpus.
Advances in neuroimaging, genomic, motion tracking, eye-tracking and many other technology-based data collection methods have led to a torrent of high dimensional datasets, which commonly have a small number of samples because of the intrinsic high cost of data collection involving human participants. High dimensional data with a small number of samples is of critical importance for identifying biomarkers and conducting feasibility and pilot work, however it can lead to biased machine learning (ML) performance estimates. Our review of studies which have applied ML to predict autistic from non-autistic individuals showed that small sample size is associated with higher reported classification accuracy. Thus, we have investigated whether this bias could be caused by the use of validation methods which do not sufficiently control overfitting. Our simulations show that K-fold Cross-Validation (CV) produces strongly biased performance estimates with small sample sizes, and the bias is still evident with sample size of 1000. Nested CV and train/test split approaches produce robust and unbiased performance estimates regardless of sample size. We also show that feature selection if performed on pooled training and testing data is contributing to bias considerably more than parameter tuning. In addition, the contribution to bias by data dimensionality, hyper-parameter space and number of CV folds was explored, and validation methods were compared with discriminable data. The results suggest how to design robust testing methodologies when working with small datasets and how to interpret the results of other studies based on what validation method was used.
Subfields: Artificial Intelligence, Biophysics, Molecular Biology
View Paper: https://www.semanticscholar.org/paper/fbf9812f29156024ec693b4633a21303eead309d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Biophysics
Citations in the corpus
- Machine Learning for the Diagnosis of Parkinson's Disease: A Review of Literature (2021, background): “Nevertheless, collecting data from a large pool of participants remains challenging in clinical studies, and data generated are commonly of high dimensionality and small sample size (Vabalas et al., 2019).”
- Evaluation of a decided sample size in machine learning applications (2023, background): “For instance, Vabalas [12] investigated the impact of a range of simulated sub-datasets (20–1000) on the performance of support vector machine (SVM) and logistic regression (LR).”
- Federated learning for medical image analysis: A survey (2023, background): “This is the well-known “small-sample-size” problem (Raudys et al., 1991; Vabalas et al., 2019).”