Correcting for Optimistic Prediction in Small Data Sets

By Gordon C. S. Smith, S. Seaman, Angela M. Wood, Patrick Royston, Ian R. White. American Journal of Epidemiology, 2014

341 citations recorded by the source corpus.

The C statistic is a commonly reported measure of screening test performance. Optimistic estimation of the C statistic is a frequent problem because of overfitting of statistical models in small data sets, and methods exist to correct for this issue. However, many studies do not use such methods, and those that do correct for optimism use diverse methods, some of which are known to be biased. We used clinical data sets (United Kingdom Down syndrome screening data from Glasgow (1991–2003), Edinburgh (1999–2003), and Cambridge (1990–2006), as well as Scottish national pregnancy discharge data (2004–2007)) to evaluate different approaches to adjustment for optimism. We found that sample splitting, cross-validation without replication, and leave-1-out cross-validation produced optimism-adjusted estimates of the C statistic that were biased and/or associated with greater absolute error than other available methods. Cross-validation with replication, bootstrapping, and a new method (leave-pair-out cross-validation) all generated unbiased optimism-adjusted estimates of the C statistic and had similar absolute errors in the clinical data set. Larger simulation studies confirmed that all 3 methods performed similarly with 10 or more events per variable, or when the C statistic was 0.9 or greater. However, with lower events per variable or lower C statistics, bootstrapping tended to be optimistic but with lower absolute and mean squared errors than both methods of cross-validation.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/dbc11bdd3601ac5e02ce94a1bac1993d79e4a46c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. J. Elharake
2. Jessica D. Ribeiro
3. S. McFadden
4. S. Omer
5. Colin G. Walsh
6. Joseph C. Franklin
7. A. Malik

In Statistics and Probability

1. Jessica D. Ribeiro
2. Colin G. Walsh
3. Joseph C. Franklin

Citations in the corpus

• Predicting Risk of Suicide Attempts Over Time Through Machine Learning (2017, methodology): “Although cross-validation and holdout sets are also viable alternatives to guard against overfitting, bootstrap optimism has been shown to provide more conservative estimates of model performance and lower absolute and mean squared errors (Hastie et al., 2001; Smith et al., 2014).”
• Determinants of COVID-19 vaccine acceptance in the US (2020, methodology): “We used bootstrap resampling (1000 samples) for internal validation and to obtain an area under the curve value accounting for model optimism.(8,9) Data were analyzed using Stata version 16 (StataCorp, College Station, Texas).”