Assessing Calibration of Prognostic Risk Scores

By Cynthia S. Crowson, Elizabeth J. Atkinson, Terry M. Therneau. 2013

282 citations recorded by the source corpus.

Current methods used to assess calibration are limited, particularly in the assessment of prognostic models. Methods for testing and visualizing calibration (e.g. the Hosmer–Lemeshow test and calibration slope) have been well thought out in the binary regression setting. However, extension of these methods to Cox models is less well known and could be improved. We describe a model-based framework for the assessment of calibration in the binary setting that provides natural extensions to the survival data setting. We show that Poisson regression models can be used to easily assess calibration in prognostic models. In addition, we show that a calibration test suggested for use in survival data has poor performance. Finally, we apply these methods to the problem of external validation of a risk score developed for the general population when assessed in a special patient population (i.e. patients with particular comorbidities, such as rheumatoid arthritis).

Subfields: Economics and Econometrics, Epidemiology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Epidemiology

In Statistics and Probability

Citations in the corpus

On Fairness and Calibration (2017, background): “It is commonly accepted amongst practitioners that both classifiers h1 and h2 should be calibrated with respect to groups G1 and G2 to prevent discrimination [4, 10, 12, 16].”
Verified Uncertainty Calibration (2019, background): “Calibration, including the squared calibration error, has been studied in many fields besides machine learning including meteorology [2, 3, 4, 5, 6], fairness [29, 30], healthcare [1, 31, 32, 33], reinforcement learning [34], natural language processing [7, 8], speech recognition [35], econometrics ”
Measuring Calibration in Deep Learning (2019, background): “The reliability of a machine learning model’s confidence in its predictions is critical for high risk applications, such as deciding whether to trust a medical diagnosis prediction (Crowson et al., 2016; Jiang et al., 2011; Raghu et al., 2018).”
The residual‐based predictiveness curve: A visual tool to assess the performance of prediction models (2016, methodology): “This strategy gives rise to the Hosmer–Lemeshow statistic (e.g., Crowson et al., 2014).”
Fairness in Recommendation Ranking through Pairwise Comparisons (2019, background): “The machine learning fairness community has primarily focused on fairness in classification, with a myriad of definitions being proposed [19, 13, 23, 17].”
Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings (2018, background): “Such lack of calibration is a serious problem in applications like medical diagnosis (Caruana et al., 2015; Crowson et al., 2016; Jiang et al., 2012), obstacle detection in self-driving vehicles (Bojarski et al., 2016), and other applications where 1Department of Computer Science and Engineering,…”
Coarse race data conceals disparities in clinical risk score performance (2023, background): “Calibration assesses how well predicted risk probabilities match the true probability of an outcome, and it is widely studied in ML and healthcare (Crowson et al., 2016; Kleinberg et al., 2016; Nixon et al., 2020; Yadlowsky et al., 2019; Deniffel et al., 2020; Khurshid et al., 2022).”
Inherent Trade-Offs in the Fair Determination of Risk Scores (2016, background): “A first basic goal in this literature is that the probability estimates provided by the algorithm should be well-calibrated: if the algorithm identifies a set of people as having a probability z of constituting positive instances, then approximately a z fraction of this set should indeed be positive”