Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews

By Rebecca M. Turner, J. Davey, Mike Clarke, S. Thompson, Julian P. T. Higgins. International Journal of Epidemiology, 2012

790 citations recorded by the source corpus.

Background Many meta-analyses contain only a small number of studies, which makes it difficult to estimate the extent of between-study heterogeneity. Bayesian meta-analysis allows incorporation of external evidence on heterogeneity, and offers advantages over conventional random-effects meta-analysis. To assist in this, we provide empirical evidence on the likely extent of heterogeneity in particular areas of health care. Methods Our analyses included 14 886 meta-analyses from the Cochrane Database of Systematic Reviews. We classified each meta-analysis according to the type of outcome, type of intervention comparison and medical specialty. By modelling the study data from all meta-analyses simultaneously, using the log odds ratio scale, we investigated the impact of meta-analysis characteristics on the underlying between-study heterogeneity variance. Predictive distributions were obtained for the heterogeneity expected in future meta-analyses. Results Between-study heterogeneity variances for meta-analyses in which the outcome was all-cause mortality were found to be on average 17% (95% CI 10–26) of variances for other outcomes. In meta-analyses comparing two active pharmacological interventions, heterogeneity was on average 75% (95% CI 58–95) of variances for non-pharmacological interventions. Meta-analysis size was found to have only a small effect on heterogeneity. Predictive distributions are presented for nine different settings, defined by type of outcome and type of intervention comparison. For example, for a planned meta-analysis comparing a pharmacological intervention against placebo or control with a subjectively measured outcome, the predictive distribution for heterogeneity is a log-normal (−2.13, 1.582) distribution, which has a median value of 0.12. In an example of meta-analysis of six studies, incorporating external evidence led to a smaller heterogeneity estimate and a narrower confidence interval for the combined intervention effect. Conclusions Meta-analysis characteristics were strongly associated with the degree of between-study heterogeneity, and predictive distributions for heterogeneity differed substantially across settings. The informative priors provided will be very beneficial in future meta-analyses including few studies.

Subfields: Public Health, Environmental and Occupational Health, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/54b716766359a3de794b16f838bddf5adfd4643d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Public Health, Environmental and Occupational Health

In Statistics and Probability

Citations in the corpus

The heterogeneity statistic I2 can be biased in small meta-analyses (2015, background): “These biases of 12 to 28 percentage points are not trivial when one considers that, in the Cochrane Library, the median I2 value is just 21% [2].”
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis† (2015, background): “2016, 7 55–79 6 8 for the between-study variance when meta-analysing (log) odds ratios (Pullenayegum, 2011; Turner et al., 2012) and standardised mean differences (Rhodes et al., 2015), and these might considerably improve estimation when few studies are included in the meta-analysis.”
The heterogeneity statistic I 2 can be biased in small meta-analyses (2015, background): “With K = 3 studies (which is the median in one summary of the Cochrane Library [22]), the bias is undefined because E(I) is only defined if df > 2.”
Trial Sequential Analysis in systematic reviews with meta-analysis (2017, background): “, a diversity of 25% or the upper confidence interval of the I(2) (provided by the TSA program)) when the required information size is calculated [48, 49].”
Automated generation of node‐splitting models for assessment of inconsistency in network meta‐analysis (2015, methodology): “The predictive distributions derived in that paper can be used as informative priors for the variance parameter σ(2) (Turner et al., 2012).”
CINeMA: Software for semiautomated assessment of the confidence in the results of network meta‐analysis (2020, neutral): “However, their critical appraisal may lead to changing the automatically generated judgments manually. boxes are updated to include extra information on heterogeneity (Turner et al., 2012).”
CINeMA: An approach for assessing confidence in the results of a network meta-analysis (2020, background): “analysed many meta-analyses of binary and continuous outcomes, categorised them according to the outcome and type of intervention and comparison, and derived empirical distributions of heterogeneity values [48,49].”
Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis (2014, methodology): “In our previous work [7], the extent to which outcomes were objectively or subjectively measured was used as a predictor, and outcome types were categorised into the broad groupings of ‘Allcause mortality’ (considered the only objectively measured outcome), ‘Semi-objective’ and ‘Subjective’ outcomes”
Robust meta‐analytic‐predictive priors in clinical trials with historical control information (2014, background): “Alternatively, a prior on τ may be derived by considering information from related patient populations or similar diseases (Higgins and Whitehead, 1996; Turner et al., 2012).”