Assessing the performance of prediction models: a framework for some traditional and novel measures

By Ewout W. Steyerberg, Andrew J. Vickers, Nancy R. Cook, Thomas Alexander Gerds, Mithat Gonen, Nancy A. Obuchowski, Michael J. Pencina, Michael W. Kattan. Epidemiology, 2010

4,522 citations recorded by the source corpus.

The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic [ROC] curve), and goodness-of-fit statistics for calibration. Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions. We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation). We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.

Subfields: Economics and Econometrics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/186cfdbacd46548855406608a6a6d560343d4370

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting (2014, background): “Performance is typically characterised by evaluating a model’s calibration and discrimination [15].”
Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers (2010, neutral): “[27].”
Early detection of cancer (2022, neutral): “It is crucial that risk models are evaluated using the appropriate methodologies (informed by statistical experts) and validated in independent data sets ( 40 ).”
Assessing Calibration of Prognostic Risk Scores (2013, methodology): “For these reasons, the calibration slope is more informative and is the preferred method for assessing calibration.(2) It might be more prevalent were it not for the prominence of the Hosmer–Lemeshow test in many statistical packages.”
The Impact of Automated Parameter Optimization on Defect Prediction Models (2018, neutral): “[129], [130].”
A new framework to enhance the interpretation of external validation studies of clinical prediction models. (2015, neutral): “Clinical usefulness often implies a threshold value for the predicted risk or probability above and below which patients are classified and differently managed [3,4,6,10,27,54].”
Firth's logistic regression with rare events: accurate effect estimates and predictions? (2017, methodology): “Furthermore, we assessed discrimination, quantified by the c-statistic, and the calibration slope of the models [10].”
Developing prediction models for clinical use using logistic regression: an overview. (2019, methodology): “In addition to the c-index, the discrimination slope is a useful and simple measure for how well the individuals who develop the outcome are separated from those who remain event-free (18).”