Sample size for binary logistic prediction models: Beyond events per variable criteria

By M. van Smeden, Karel G. M. Moons, Joris A. H. de Groot, Gary S. Collins, Douglas G. Altman, Marinus J. C. Eijkemans, Johannes B Reitsma. Statistical Methods in Medical Research, 2018

679 citations recorded by the source corpus.

Binary logistic regression is one of the most frequently applied statistical approaches for developing clinical prediction models. Developers of such models often rely on an Events Per Variable criterion (EPV), notably EPV ≥10, to determine the minimal sample size required and the maximum number of candidate predictors that can be examined. We present an extensive simulation study in which we studied the influence of EPV, events fraction, number of candidate predictors, the correlations and distributions of candidate predictor variables, area under the ROC curve, and predictor effects on out-of-sample predictive performance of prediction models. The out-of-sample performance (calibration, discrimination and probability prediction error) of developed prediction models was studied before and after regression shrinkage and variable selection. The results indicate that EPV does not have a strong relation with metrics of predictive performance, and is not an appropriate criterion for (binary) prediction model development studies. We show that out-of-sample predictive performance can better be approximated by considering the number of predictors, the total sample size and the events fraction. We propose that the development of new sample size criteria for prediction models should be based on these three parameters, and provide suggestions for improving sample size determination.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/b4d769e4a62cd4252631c976b8ca5655cd792449

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. (2019, background): “Recent research clearly indicated that this increases the risk of overfitting [103].”
TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods (2024, background): “…of TRIPOD 2015, there have been numerous methodological advances in prediction modelling, including sample size guidance for developing models 23-27 and evaluating their performance, 28-32 and greater recognition of operationalising fairness, 33 reproducibility, 34 and adopting open science…”
Calibration: the Achilles heel of predictive analytics (2019, background): “It is important to prespecify the modeling strategy and to ensure that sample size is sufficient for the number of considered predictors [30, 31].”