Cross‐validation failure: Small sample sizes lead to large error bars
By Gaël Varoquaux. NeuroImage, 2017
697 citations recorded by the source corpus.
ABSTRACT Predictive models ground many state‐of‐the‐art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross‐validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross‐validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg Symbol for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data. Symbol. No caption available.
View Paper: https://www.semanticscholar.org/paper/ccf0099a72593d734c2cf441d8f4909e76c5f929
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Simon B. Eickhoff
- Danilo Bzdok
- Habib Benali
- Ovidiu V. Lungu
- Ella Gabitov
- Basile Pinsard
- Bert Heinrichs
- Arnaud Boutin
Citations in the corpus
- Machine Learning for Precision Psychiatry: Opportunities and Challenges. (2017, background): “Small sample sizes exacerbate the tendency of adaptive models learning noise in the data - overfitting (44).”
- Consolidation alters motor sequence-specific distributed representations (2018, methodology): “However, the classification-based measures they 112 used may have biased their parametric statistical results by violating both the normality 113 assumption and theoretical null-distribution (Allefeld et al. 2015; Combrisson and Jerbi 114 2015; Jamalabadi et al. 2016; Varoquaux 2017) and may have th”
- Machine learning algorithm validation with a limited sample size (2019, methodology): “Arbabshirani et al. [3] and Varoquaux [4] reviewed studies, which applied ML methods for prediction of various brain disorders and found that studies with smaller sample sizes tended to report higher classification performance.”
- Benchmarking functional connectome‐based predictive models for resting‐state fMRI (2019, unknown): “Recommendations on methods that perform well can increase practitioner's productivity and limit vibration effects that risk undermining the reliability of biomarkers (Varoquaux, 2017).”
- Your evidence? Machine learning algorithms for medical diagnosis and prediction (2019, background): “At the moment, such approach is clearly limited by data availability, leading to a dominance of cross-validation over external evaluation (Mendelson, Zuluaga, Lorenzi, Hutton, & Ourselin, 2016; Varoquaux, 2018; Whelan & Garavan, 2014).”