Evidence of bias and variation in diagnostic accuracy studies

By A. Rutjes, Johannes B Reitsma, M. Di Nisio, N. Smidt, J. C. van Rijn, Patrick M. M. Bossuyt. Canadian Medical Association Journal, 2006

635 citations recorded by the source corpus.

Background: Studies with methodologic shortcomings can overestimate the accuracy of a medical test. We sought to determine and compare the direction and magnitude of the effects of a number of potential sources of bias and variation in studies on estimates of diagnostic accuracy. Methods: We identified meta-analyses of the diagnostic accuracy of tests through an electronic search of the databases MEDLINE, EMBASE, DARE and MEDION (1999–2002). We included meta-analyses with at least 10 primary studies without preselection based on design features. Pairs of reviewers independently extracted study characteristics and original data from the primary studies. We used a multivariable meta-epidemiologic regression model to investigate the direction and strength of the association between 15 study features on estimates of diagnostic accuracy. Results: We selected 31 meta-analyses with 487 primary studies of test evaluations. Only 1 study had no design deficiencies. The quality of reporting was poor in most of the studies. We found significantly higher estimates of diagnostic accuracy in studies with nonconsecutive inclusion of patients (relative diagnostic odds ratio [RDOR] 1.5, 95% confidence interval [CI] 1.0–2.1) and retrospective data collection (RDOR 1.6, 95% CI 1.1–2.2). The estimates were highest in studies that had severe cases and healthy controls (RDOR 4.9, 95% CI 0.6–37.3). Studies that selected patients based on whether they had been referred for the index test, rather than on clinical symptoms, produced significantly lower estimates of diagnostic accuracy (RDOR 0.5, 95% CI 0.3–0.9). The variance between meta-analyses of the effect of design features was large to moderate for type of design (cohort v. case–control), the use of composite reference standards and the use of differential verification; the variance was close to zero for the other design features. Interpretation: Shortcomings in study design can affect estimates of diagnostic accuracy, but the magnitude of the effect may vary from one situation to another. Design features and clinical characteristics of patient groups should be carefully considered by researchers when designing new studies and by readers when appraising the results of such studies. Unfortunately, incomplete reporting hampers the evaluation of potential sources of bias in diagnostic accuracy studies.

Subfields: Family Practice, Physiology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/1d2ff1b5d7bca210bd96db23030fc8c389fc2a09

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

In Statistics and Probability

Citations in the corpus

Systematic Reviews of Diagnostic Test Accuracy (2008, neutral): “Sources of bias for which unambiguous evidence indicates that they lead to overestimation of diagnostic accuracy are the inclusion of healthy control participants and the differential use of reference standards (24, 26).”
Screening for Depression in the General Population with the Center for Epidemiologic Studies Depression (CES-D): A Systematic Review with Meta-Analysis (2016, neutral): “The cut-off point of 22 was assessed in 7 of the studies, and the pooled estimate of sensitivity was 0.79 (95% CI: 0.69–0.85), specificity = 0.80 (95% CI: 0.75–0.85), with DOR = 14.7.”
A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. (2009, background): “The magnitude of the biasing effect will depend on the frequency of errors by the imperfect reference standard and the degree of correlation in errors between index test and reference standard [15,16].”
Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist (2014, background): “, blinded to) the results of the index tests is well established [35,40,54,69,79,80,85].”
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration (2016, neutral): “2 29 In the example, the authors explicitly described a convenience series where participants were enrolled based on their accessibility to the clinical investigator.”