External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

By Gary S. Collins, Joris A. H. de Groot, S. Dutton, Omar Omar, M. Shanyinde, A. Tajar, M. Voysey, R. Wharton, Ly-Mee Yu, Karel G. M. Moons and 1 more. BMC Medical Research Methodology, 2014

646 citations recorded by the source corpus.

BackgroundBefore considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.MethodsWe conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.Results11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.ConclusionsThe vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Subfields: Epidemiology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/27592b12cf12c25dbd6854356abf1e5502fd7ebd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Epidemiology

In Statistics and Probability

Citations in the corpus

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study (2015, background): “Despite the clear importance of external validation, the design requirements for studies that attempt to evaluate the performance of multivariable prognostic models in new data have been little explored [7,12,13].”
A new framework to enhance the interpretation of external validation studies of clinical prediction models. (2015, positive): “Similar guidelines on reporting are currently being developed for studies developing or validating risk prediction models [43,50].”
Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. (2015, neutral): “Several reviews have evaluated the quality of published reports that describe the development or validation prediction models [23–28].”
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement (2015, positive): “Several reviews have evaluated the quality of published reports that describe the development or validation prediction models [23-28].”
Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness (2020, background): “60 In external validation studies, reductions in the predictive accuracy of models (relative to their original performance in development studies) is expected.”
Towards better clinical prediction models: seven steps for development and an ABCD for validation. (2014, methodology): “Whatever the method used to develop a model, one could argue that validity is all that matters.(7) We propose four key measures in the assessment of the validation of prediction models, related to calibration, discrimination, and clinical usefulness.”
Calibration: the Achilles heel of predictive analytics (2019, background): “Systematic reviews have found that calibration is assessed far less often than discrimination [2, 3, 5 – 7], which is problematic since poor calibration can make predictions misleading [8].”
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015, positive): “Several reviews have evaluated the quality of published reports that describe the development or validation prediction models [23-28].”
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement (2015, positive): “…have showed that the quality of reporting in published articles describing the development or validation of multivariable prediction models in medicine is poor (Laupacis et al, 1997; Ettema et al, 2010; Mallett et al, 2010; Collins et al, 2011; Bouwmeester et al, 2012; Collins et al, 2013, 2014).”
Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review (2021, background): “…several publications have raised concern about the methodological quality of prediction models developed with conventional statistical techniques, 6 17 18 a formal methodological and risk of bias assessment of supervised machine learning based prediction model studies across all medical…”