PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

By Robert Wolff, Karel G. M. Moons, R. Riley, P. Whiting, M. Westwood, Gary S. Collins, Johannes B Reitsma, J. Kleijnen, S. Mallett. Annals of Internal Medicine, 2019

2,019 citations recorded by the source corpus.

Prediction relates to estimating the probability of something currently unknown. In the context of medical research, prediction typically concerns either diagnosis (probability of a certain condition being present but not yet detected) or prognosis (probability of an outcome developing in the future) (13). Prognosis applies not only to sick persons or those with an established diagnosis but also to, for example, pregnant women at risk for diabetes (4). Prediction research includes predictor finding studies, prediction model studies (development, validation, and extending or updating), and prediction model impact studies (1). Predictor finding studies (also known as risk factor or prognostic factor studies) aim to identify which predictors (such as age, disease stage, or biomarkers) independently contribute to the prediction of a diagnostic or prognostic outcome (1, 5). Prediction model studies aim to develop, validate, or update (for example, extend) a multivariable prediction model. A prediction model uses multiple predictors in combination to estimate probabilities to inform and often guide individual care (2, 6, 7). These models can predict an individual's probability of either currently having a particular outcome or disease (diagnostic prediction model) or having a particular outcome in the future (prognostic prediction model). Both types of model are widely used in various medical domains and settings (810), as evidenced by the large number of models developed in cancer (11, 12), neurology (13, 14), and cardiovascular disease (15). Prediction models are sometimes described as risk prediction models, predictive models, prediction indices or rules, or risk scores (2, 7). An example is QRISK2 for predicting cardiovascular risk (16). Prediction model impact studies evaluate the effect of using a model to guide patient care compared with not using such a model. They use a comparative design, such as a randomized trial, to study the model's effect on clinical decision making, patient outcomes, or costs of care (1). Systematic reviews have a key role in evidence-based medicine and the development of clinical guidelines (1719). They are considered to provide the most reliable form of evidence for the effects of an intervention or diagnostic test (20, 21). Systematic reviews of prediction models are a relatively new and evolving area but are increasingly undertaken to systematically identify, appraise, and summarize evidence on the performance of prediction models (1, 6, 22). Assessing the quality of included studies is a crucial step in any systematic review (20, 21). The QUIPS (Quality In Prognosis Studies) tool has been developed to assess risk of bias (ROB) in predictor finding (prognostic factor) studies (23). Researchers can use the revised Cochrane ROB Tool (ROB 2.0) (24) to investigate the methodological quality of prediction model impact studies that use a randomized comparative design, or ROBINS-I (Risk Of Bias In Nonrandomized Studies of Interventions) for those that use a nonrandomized comparative design (25). As more prediction model studies and systematic reviews of such studies are used as evidence for clinical guidance, a tool facilitating quality assessment for individual prediction model studies is urgently needed. We present PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool to assess the ROB and concerns regarding the applicability of diagnostic and prognostic prediction model studies. PROBAST can be used to assess studies of model development and model validation, including those updating a prediction model (Box 1 [26]). We refer to the accompanying explanation and elaboration document (27), for detailed explanations of how to use PROBAST and how to judge ROB and applicability. Box 1. Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST. Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (7, 26). PROBAST = Prediction model Risk Of Bias ASsessment Tool. Methods: Development of PROBAST Development of PROBAST was based on a 4-stage approach for developing health research reporting guidelines: define the scope, review the evidence base, use a Web-based Delphi procedure, and refine the tool through piloting (28). Guidelines explicitly aimed at the development of quality assessment tools were not available at the time (29). Development Stage 1: Scope and Definitions A steering group of 9 experts in prediction model studies and development of quality assessment tools agreed on key features of the desired scope of PROBAST. A panel of 38 experts with different backgrounds further refined the scope during the Web-based Delphi procedure. PROBAST was designed mainly to assess primary studies included in a systematic review. The group agreed that PROBAST would assess both risk of bias and concerns regarding applicability of a study evaluating a multivariable prediction model to be used for individualized diagnosis or prognosis. A domain-based structure was adopted, similar to that used in other ROB tools, such as ROB 2.0 (24), ROBINS-I (25), QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) (30), and ROBIS (31). We agreed that PROBAST should cover primary studies that develop, validate, or update multivariable prediction models aiming to make individualized predictions of a diagnostic or prognostic outcome (Box 1). Studies that use multivariable modeling techniques to identify predictors (such as risk or prognostic factors) associated with an outcome but do not attempt to develop, validate, or update a model for making individualized predictions are not covered by PROBAST (5). Therefore, PROBAST is not intended for predictor finding studies or prediction model impact studies. Studies of diagnostic and prognostic models often use different terms for predictors and outcomes (Box 2). A multivariable prediction model is defined as any combination or equation of 2 or more predictors for estimating probability or risk for an individual (6, 7, 3234). Box 2. Differences between diagnostic and prognostic prediction model studies. PROBAST = Prediction model Risk Of Bias ASsessment Tool. Development Stage 2: Review of Evidence We used the following 3 approaches to build an evidence base to inform the development of PROBAST: identifying relevant methodological reviews in the area of prediction model research (November 2012 to January 2013), asking members of the steering group to identify relevant methodological studies (January 2013 to March 2013), and using the Delphi procedure to ask members of the wider group to identify additional evidence (February 2012 to July 2014). Identified literature was used to guide the scope and produce an initial list of signaling questions to consider for inclusion in PROBAST (1, 2, 57, 26, 3340). We grouped signaling questions into common themes to identify possible domains. Additional literature provided as part of the Web-based surveys informed development of the explanation and elaboration document. Development Stage 3: Web-Based Delphi Procedure We used a modified Delphi process to gain feedback and agreement on the scope, structure, and content of PROBAST. Web-based surveys were developed to gather structured feedback for each round. The 38-member Delphi group comprised methodological experts in prediction model research and development of quality assessment tools, experienced systematic reviewers, commissioners, and representatives of reimbursement agencies. We included various stakeholders to ensure that the views of end users, methodological experts, and decision makers were represented. The Delphi process consisted of 7 rounds. Round 1 asked about the scope of the tool, and participants agreed to focus on prediction model studies and follow a domain-based structure. Round 2 aimed to identify relevant domains and agree on which to include. The signaling questions for domains were refined in rounds 3 to 5. Respondents used a 1-to-5 Likert scale to rate each proposed signaling question for inclusion. They could also suggest rephrasing, provide supporting evidence (such as references to relevant studies), and suggest missing signaling questions. Round 6 refined the domains and introduced further optional guidance for using PROBAST. In the last round, participants received the agreed draft version of PROBAST and had the opportunity to provide any final feedback. Development Stage 4: Piloting and Refining the Tool We held 6 workshops on PROBAST at consecutive annual Cochrane Colloquia (Quebec, Canada, 2013; Hyderabad, India, 2014; Vienna, Austria, 2015; Seoul, South Korea, 2016; Cape Town, South Africa, 2017; and Edinburgh, United Kingdom, 2018). We also held numerous consecutive workshops with MSc and PhD students (for example, the master's program in epidemiology at Utrecht University [Utrecht, the Netherlands] and the Evidence-Based Health Care program at Oxford University [Oxford, United Kingdom]). In these workshops, we piloted the then-current version of PROBAST to gather feedback on practical issues associated with using the tool so that we could further refine and subsequently validate it. Finally, more than 50 review groups have already piloted PROBAST versions, including the final version, in their reviews. Topics included cancer, cardiology, endocrinology, pulmonology, and orthopedics. All feedback received from these initiatives was used to further inform the content and structure of PROBAST, wording of the signaling questions, and content of the guidance documents (27). Results: The PROBAST Tool What Does PROBAST Assess? PROBAST assesses both risk of bias and concerns regarding applicability of primary studies that developed or validated multivariable prediction models for diagnosis or prognosis (Boxes 1 and 2).

Subfields: Economics and Econometrics, Public Health, Environmental and Occupational Health, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/b52b38ffdfdcb6253a99731118caf1d69c90b9b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Public Health, Environmental and Occupational Health

In Statistics and Probability

Citations in the corpus

External validation of prognostic models: what, why, how, when and where? (2020, methodology): “Bias in prognostic models can be assessed using the Prediction model Risk Of Bias ASsessment Tool tool [31].”
Bias in medical AI: Implications for clinical decision-making (2024, methodology): “…Risk of Bias Assessment (PROBAST) tool have been developed to assess the risk of imbalanced data biases, which allows both developers and downstream users (e.g., physicians, hospitals) to better understand the suitability and limitations of AI-based systems in specific clinical settings [54–56].”
Developing clinical prediction models: a step-by-step guide (2024, positive): “10-14 PROBAST (prediction model risk-of-bias assessment tool) provides a structured way to assess the risk of bias in a prediction modelling study.”
Fairness of artificial intelligence in healthcare: review and recommendations (2023, background): “This common scenario means that estimations of AI performance based on internal test data may overestimate its real‑world performance on external data [73–75].”
Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal (2020, neutral): “231 The PRIEST score 262 predicts 30 day death or organ support in patients with suspected or confirmed covid-19 presenting at the emergency department.”
Bias in artificial intelligence algorithms and recommendations for mitigation (2023, methodology): “A number of comprehensive frameworks and checklists have been developed, such as the Translational Evaluation of Healthcare AI (TEHAI) [46], the DECIDE-AI [47], the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) [48], the prediction model risk of bias assessment tool”