Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve.

By Nancy R. Cook. Clinical Chemistry, 2008

699 citations recorded by the source corpus.

BACKGROUND Diagnostic and prognostic or predictive models serve different purposes. Whereas diagnostic models are usually used for classification, prognostic models incorporate the dimension of time, adding a stochastic element. CONTENT The ROC curve is typically used to evaluate clinical utility for both diagnostic and prognostic models. This curve assesses how well a test or model discriminates, or separates individuals into two classes, such as diseased and nondiseased. A strong risk predictor, such as lipids for cardiovascular disease, may have limited impact on the area under the curve, called the AUC or c-statistic, even if it alters predicted values. Calibration, measuring whether predicted probabilities agree with observed proportions, is another component of model accuracy important to assess. Reclassification can directly compare the clinical impact of two models by determining how many individuals would be reclassified into clinically relevant risk strata. For example, adding high-sensitivity C-reactive protein and family history to prediction models for cardiovascular disease using traditional risk factors moves approximately 30% of those at intermediate risk levels, such as 5%-10% or 10%-20% 10-year risk, into higher or lower risk categories, despite little change in the c-statistic. A calibration statistic can asses how well the new predicted values agree with those observed in the cross-classified data. SUMMARY Although it is useful for classification, evaluation of prognostic models should not rely solely on the ROC curve, but should assess both discrimination and calibration. Risk reclassification can aid in comparing the clinical impact of two models on risk for the individual, as well as the population.

Subfields: Cardiology and Cardiovascular Medicine, Epidemiology

View Paper: https://www.semanticscholar.org/paper/1116cfe2775c806d4821957843eb7f5ddf8d450a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cardiology and Cardiovascular Medicine

In Epidemiology

Citations in the corpus

Predictive validity performance indicators in violence risk assessment: a methodological primer. (2013, methodology): “ROC curve analysis was developed as a diagnostic methodology as opposed to a prognostic methodology (Cook, 2008).”
Development and Validation of a Clostridium difficile Infection Risk Prediction Model (2011, methodology): “These calculations are rudimentary and assume that the interventions will be 100% effective at preventing CDI.”
Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist (2014, positive): “The c-statistic should not be used as the only performance measure, however, since it is influenced by the distribution of predictor values and is often insensitive to inclusion of an additional predictor in the model [59,122–125].”
External validation of prognostic models: what, why, how, when and where? (2020, methodology): “Good discrimination is important, as it enables the model to correctly classify patients into risk groups [26].”
Ensuring Fairness in Machine Learning to Advance Health Equity (2018, methodology): “The quality of a model is assessed by computing evaluation metrics on data not used to build the model, such as sensitivity, specificity, or the c-statistic, which measures the ability of a model to distinguish patients with a condition from those without it (19, 20).”
Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review (2017, methodology): “Because the value of the c -statistic is independent of the prevalence of the outcome, 133 it is useful for comparing models across different diseases.”
Assessing the performance of prediction models: a framework for some traditional and novel measures (2010, methodology): “Cook proposed a reclassification test as a variant of the Hosmer-Lemeshow statistic within the reclassified categories, leading to a 2 statistic.(44) Pencina et al(9) has extended the reclassification idea by conditioning on the outcome: reclassification of subjects with and without the outcome shou”
Prognostic Models With Competing Risks: Methods and Application to Coronary Risk Prediction (2009, result): “Second, the c index is relatively insensitive to small but potentially clinically-relevant changes in predictive accuracy.(34,35) This could be the reason why the c index, but not calibration, is virtually identical for all 3 models in our case study.”