External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges

By R. Riley, J. Ensor, Kym I. E. Snell, T. Debray, Douglas G. Altman, Karel G. M. Moons, Gary S. Collins. 2016

510 citations recorded by the source corpus.

Access to big datasets from e-health records and individual participant data (IPD) meta-analysis is signalling a new advent of external validation studies for clinical prediction models. In this article, the authors illustrate novel opportunities for external validation in big, combined datasets, while drawing attention to methodological challenges and reporting issues.

Subfields: Artificial Intelligence, Cardiology and Cardiovascular Medicine, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cardiology and Cardiovascular Medicine

In Statistics and Probability

Citations in the corpus

Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, background): “One possible explanation for any nonlinear relationship is that the risk score was well-calibrated on its original training data, but the audit data has a different distribution (data shift) [29].”
MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care (2020, background): “It is clear that the performance of any AI model broadly depends on its reliability and its ability to generalize to the setting and population in which it is applied, rather than its performance represented by the training and test data alone.(10) However, the characteristics of the data necessary ”
The myth of generalisability in clinical research and machine learning in health care (2020, background): “30,31 Finally, we note that there are some circumstances where broad generalisability is desirable.”
Fairness Violations and Mitigation under Covariate Shift (2019, methodology): “We artificially introduce two types of shifts – (a) change in female proportion, and (b) change in measurement policy, where a lab test is prescribed less often – some of the factors affecting model performance across clinical settings [60].”
Calibration: the Achilles heel of predictive analytics (2019, background): “For example, university hospitals may treat more patients with the event of interest than regional hospitals; such heterogeneity between settings can affect risk estimates and their calibration [18].”
External validation of prognostic models: what, why, how, when and where? (2020, background): “Finally, differences in case mix between the development and validation cohorts can significantly influence predictive performance, even if the predictor effects are the same [36].”
A tutorial on calibration measurements and calibration models for clinical prediction models (2020, background): “In a relatively recent article from the biomedical informatics literature, Walsh et al 31,32 re-emphasize the calls from Van Calster et al 33 and Riley et al 34 on the importance of calibration in addition to discrimination when evaluating predictive models.”