Calibration: the Achilles heel of predictive analytics
By B. van Calster, D. McLernon, Maarten van Smeden, Laure Wynants, E. Steyerberg, Patrick Gary S. Petra David J. Karel G. M. Ewout W. Ben Ma Bossuyt Collins Macaskill McLernon Moons Steyerber, Patrick M. M. Bossuyt, Gary S. Collins, Petra Macaskill, Karel G. M. Moons and 1 more. BMC Medicine, 2019
1,657 citations recorded by the source corpus.
BackgroundThe assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.Main textHerein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.ConclusionEfforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.
Subfields: Artificial Intelligence, Epidemiology, Health Informatics
View Paper: https://www.semanticscholar.org/paper/2cb777cd1965f77102da963a69ca748d4ff6bd5e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Johannes B Reitsma
- Karel G. M. Moons
- T. Debray
- M. van Smeden
- Lotty Hooft
- Steven W. J. Nijman
- E. Schuit
- Ewout W. Steyerberg
In Epidemiology
- Johannes B Reitsma
- Ewout W. Steyerberg
- T. Debray
- Karel G. M. Moons
- Lotty Hooft
- Douglas G. Altman
- Gary S. Collins
In Health Informatics
Citations in the corpus
- Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022, methodology): “Still, AIPM updating, site-speci fi c training or recalibration might be needed to adapt an existing AIPM to a different healthcare setting 3,5,9,60,68,72 .”
- Real-time prediction of COVID-19 related mortality using electronic health records (2020, neutral): “In addition to assessing predictive performance, we also evaluated the calibration [23] of the risk scores predicted by CovEWS.”
- Comparison of Conventional Statistical Methods with Machine Learning in Medicine: Diagnosis, Drug Development, and Treatment (2020, background): “Possibly much bigger sample sizes are required to gain reliable (calibrated) risk predictions [44] than reliable (diagnostic) classifications.”