Calibrating predictive model estimates to support personalized medicine

By Xiaoqian Jiang, Melanie Osl, Jihoon Kim, Lucila Ohno-Machado. 2011

250 citations recorded by the source corpus.

Objective Predictive models that generate individualized estimates for medically relevant outcomes are playing increasing roles in clinical care and translational research. However, current methods for calibrating these estimates lose valuable information. Our goal is to develop a new calibration method to conserve as much information as possible, and would compare favorably to existing methods in terms of important performance measures: discrimination and calibration. Material and methods We propose an adaptive technique that utilizes individualized confidence intervals (CIs) to calibrate predictions. We evaluate this new method, adaptive calibration of predictions (ACP), in artificial and real-world medical classification problems, in terms of areas under the ROC curves, the Hosmer-Lemeshow goodness-of-fit test, mean squared error, and computational complexity. Results ACP compared favorably to other calibration methods such as binning, Platt scaling, and isotonic regression. In several experiments, binning, isotonic regression, and Platt scaling failed to improve the calibration of a logistic regression model, whereas ACP consistently improved the calibration while maintaining the same discrimination or even improving it in some experiments. In addition, the ACP algorithm is not computationally expensive. Limitations The calculation of CIs for individual predictions may be cumbersome for certain predictive models. ACP is not completely parameter-free: the length of the CI employed may affect its results. Conclusions ACP can generate estimates that may be more suitable for individualized predictions than estimates that are calibrated using existing methods. Further studies are necessary to explore the limitations of ACP.

Subfields: Artificial Intelligence, Epidemiology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Epidemiology

Citations in the corpus

Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “This is critical in tasks like pedestrian detection for self-driving cars [21] or automated diagnosis of medical conditions [22].”
Verified Uncertainty Calibration (2019, background): “Calibration, including the squared calibration error, has been studied in many fields besides machine learning including meteorology [2, 3, 4, 5, 6], fairness [29, 30], healthcare [1, 31, 32, 33], reinforcement learning [34], natural language processing [7, 8], speech recognition [35], econometrics ”
Measuring Calibration in Deep Learning (2019, background): “The reliability of a machine learning model’s confidence in its predictions is critical for high risk applications, such as deciding whether to trust a medical diagnosis prediction (Crowson et al., 2016; Jiang et al., 2011; Raghu et al., 2018).”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, background): “Calibrated confidence estimates have been extensively studied by practitioners in medicine (Jiang et al., 2012), meteorology (Raftery et al.”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “Calibration Calibration is a well-studied topic in other tasks such as medical diagnosis (Jiang et al., 2012) and image recognition (Guo et al., 2017; Lee et al., 2018).”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): “Accurate estimates of uncertainty are important for difficult or sensitive prediction problems that have variable accuracy (Amodei et al., 2016; Jiang et al., 2012; 2018; Angelopoulos et al., 2021).”
Calibrated Structured Prediction (2015, background): “Its importance has been discussed and emphasized in medicine [22], natural language processing [23], speech recognition [21], meteorology [10], econometrics [9], and psychology [24].”
Calibration of Pre-trained Transformers (2020, background): “One step towards understanding whether these models can be trusted is by analyzing whether they are calibrated (Raftery et al., 2005; Jiang et al., 2012; Kendall and Gal, 2017): how aligned their posterior probabilities are with empirical likelihoods (Brier, 1950; Guo et al., 2017).”
Revisiting the Calibration of Modern Neural Networks (2021, methodology): “, 2016), medical diagnosis (Esteva et al., 2017; Jiang et al., 2012), and meteorological forecasting (Sønderby et al.”
Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings (2018, background): “Such lack of calibration is a serious problem in applications like medical diagnosis (Caruana et al., 2015; Crowson et al., 2016; Jiang et al., 2012), obstacle detection in self-driving vehicles (Bojarski et al., 2016), and other applications where 1Department of Computer Science and Engineering,…”
On Calibration of Modern Neural Networks (2017, background): “Alternatively, in automated health care, control should be passed on to human doctors when the confidence of a disease diagnosis network is low (Jiang et al., 2012).”
Binary Classifier Calibration: Bayesian Non-Parametric Approach (2014, result): “The results show the methods either outperform or are comparable in performance to the state-of-the-art calibration methods.”
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift (2022, background): “…has been widely studied in machine learning [Naeini et al., 2014, Guo et al., 2017, Kumar et al., 2019], and applications such as meteorology [Murphy, 1973, DeGroot and Fienberg, 1983, Gneiting and Raftery, 2005], fairness [Hebert-Johnson et al., 2018], and health-care [Jiang et al., 2012].”