A new framework to enhance the interpretation of external validation studies of clinical prediction models.

By T. Debray, Y. Vergouwe, Hendrik Koffijberg, D. Nieboer, Ewout W. Steyerberg, Karel G. M. Moons. Journal of Clinical Epidemiology, 2015

506 citations recorded by the source corpus.

OBJECTIVES It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models. STUDY DESIGN AND SETTING We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting. RESULTS We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings. CONCLUSION The proposed framework enhances the interpretation of findings at external validation of prediction models.

Subfields: Artificial Intelligence, Internal Medicine, Pulmonary and Respiratory Medicine

View Paper: https://www.semanticscholar.org/paper/73bfc8cfd1845925c204dbbb8478d486f7fdae3d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Pulmonary and Respiratory Medicine

Citations in the corpus

External validation of prognostic models: what, why, how, when and where? (2020, methodology): “If individual participant data from development and validation cohorts are available, more advanced statistical approaches have been developed to calculate an overall measure of similarity between cohorts [18].”
External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges (2016, background): “20 Recalibration is a form of model updating, where particular components of the developed model (such as the intercept or base-line hazard rate, or even particular predictor effects) are modified or tailored for each study population of interest.”
Prediction models need appropriate internal, internal-external, and external validation (2015, unknown): “The similarity of validation to development sets hence is essential for interpretation of an external validation study, either by comparing descriptive data (‘‘Table 1’’) or a simple statistical model to predict membership of the development or validation data set [16].”
Prediction models for cardiovascular disease risk in the general population: systematic review (2016, unknown): “62 63 Analysis of such combined or large datasets has the advantage not only of increased total sample size, but also of better tackling case mix effects, setting specific issues (eg, inclusion of setting specific predictors), and better tailoring of existing models to different settings and…”
Sample size considerations for the external validation of a multivariable prognostic model: a resampling study (2015, unknown): “23 case-mix is different or the underlying population in the validation data set is different to that from which the model was originally developed [50].”
Key challenges for delivering clinical impact with artificial intelligence (2019, background): “This will ensure that all relevant variations in patient demographics and disease states of target patients in real-world clinical settings are adequately represented in the system where it will be applied [76].”