Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

By Peter C. Austin, Ewout W. Steyerberg. Statistics in Medicine, 2013

345 citations recorded by the source corpus.

Predicting the probability of the occurrence of a binary outcome or condition is important in biomedical research. While assessing discrimination is an essential issue in developing and validating binary prediction models, less attention has been paid to methods for assessing model calibration. Calibration refers to the degree of agreement between observed and predicted probabilities and is often assessed by testing for lack‐of‐fit. The objective of our study was to examine the ability of graphical methods to assess the calibration of logistic regression models. We examined lack of internal calibration, which was related to misspecification of the logistic regression model, and external calibration, which was related to an overfit model or to shrinkage of the linear predictor. We conducted an extensive set of Monte Carlo simulations with a locally weighted least squares regression smoother (i.e., the loess algorithm) to examine the ability of graphical methods to assess model calibration. We found that loess‐based methods were able to provide evidence of moderate departures from linearity and indicate omission of a moderately strong interaction. Misspecification of the link function was harder to detect. Visual patterns were clearer with higher sample sizes, higher incidence of the outcome, or higher discrimination. Loess‐based methods were also able to identify the lack of calibration in external validation samples when an overfit regression model had been used. In conclusion, loess‐based smoothing methods are adequate tools to graphically assess calibration and merit wider application. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons, Ltd

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/d3988c8fbf5b8434b74fbc401b32ff6942fef3b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. (2019, methodology): “Preferably, calibration should be investigated using calibration curves, whereas the Hosmer-Lemeshow test should be avoided [18,98,99].”
A new test and graphical tool to assess the goodness of fit of logistic regression models (2016, methodology): “To support the interpretation of possible lacks of fit, graphical and qualitative methods have also been proposed [5].”
Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records. (2020, neutral): “28,29 We adopted the usual approach for binary outcomes of plotting means of decile-binned predictions on the x -axis and means of the observed outcomes in the patients in each bin on the y -axis.”
Towards better clinical prediction models: seven steps for development and an ABCD for validation. (2014, methodology): “values, smoothing techniques can well be used to depict the association.(37) We can also plot the observed proportions of death for”
Calibration: the Achilles heel of predictive analytics (2019, methodology): “The concepts explained in this section are illustrated in Additional 1, with the validation of the Risk of Ovarian Malignancy Algorithm (ROMA) for the diagnosis of ovarian malignancy in women with an ovarian tumor selected for surgical removal [28]; further details can be found elsewhere [1, 4, 29].”