Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes

By R. Riley, Kym I. E. Snell, J. Ensor, D. Burke, Frank E. Harrell, Karel G. M. Moons, Gary S. Collins. Statistics in Medicine, 2018

1,038 citations recorded by the source corpus.

When designing a study to develop a new prediction model with binary or time‐to‐event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R2, and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox‐Snell R2, which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

Subfields: Economics and Econometrics

View Paper: https://www.semanticscholar.org/paper/e6d44d68ba9aeb0e76841c9ff3376a23de9b76b9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

Citations in the corpus

Calibration: the Achilles heel of predictive analytics (2019, background): “It is important to prespecify the modeling strategy and to ensure that sample size is sufficient for the number of considered predictors [30, 31].”
External validation of prognostic models: what, why, how, when and where? (2020, background): “This will result in predicted risks that are too extreme when used in new patients [19].”
TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods (2024, background): “…of TRIPOD 2015, there have been numerous methodological advances in prediction modelling, including sample size guidance for developing models 23-27 and evaluating their performance, 28-32 and greater recognition of operationalising fairness, 33 reproducibility, 34 and adopting open science…”