The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews

By Rebecca M. Turner, S. Bird, Julian P. T. Higgins. PLoS ONE, 2013

621 citations recorded by the source corpus.

Background Most meta-analyses include data from one or more small studies that, individually, do not have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored. We examine the distribution of power available in studies within meta-analyses published in Cochrane reviews, and investigate the impact of underpowered studies on meta-analysis results. Methods and Findings For 14,886 meta-analyses of binary outcomes from 1,991 Cochrane reviews, we calculated power per study within each meta-analysis. We defined adequate power as ≥50% power to detect a 30% relative risk reduction. In a subset of 1,107 meta-analyses including 5 or more studies with at least two adequately powered and at least one underpowered, results were compared with and without underpowered studies. In 10,492 (70%) of 14,886 meta-analyses, all included studies were underpowered; only 2,588 (17%) included at least two adequately powered studies. 34% of the meta-analyses themselves were adequately powered. The median of summary relative risks was 0.75 across all meta-analyses (inter-quartile range 0.55 to 0.89). In the subset examined, odds ratios in underpowered studies were 15% lower (95% CI 11% to 18%, P<0.0001) than in adequately powered studies, in meta-analyses of controlled pharmacological trials; and 12% lower (95% CI 7% to 17%, P<0.0001) in meta-analyses of controlled non-pharmacological trials. The standard error of the intervention effect increased by a median of 11% (inter-quartile range −1% to 35%) when underpowered studies were omitted; and between-study heterogeneity tended to decrease. Conclusions When at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered studies often contribute little information, and could be left out if a rapid review of the evidence is required. However, underpowered studies made up the entirety of the evidence in most Cochrane reviews.

Subfields: General Health Professions, Public Health, Environmental and Occupational Health, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/1ca137890099394abf413e7386b0336cebaa5aa8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Public Health, Environmental and Occupational Health

In Statistics and Probability

Citations in the corpus

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods (2014, background): “Most review authors do not assess if an accrued information size is sufficient or not to detect or reject a given intervention effect, and this is problematic as nearly all meta-analyses in systematic reviews are underpowered (the meta-analyses do not have enough information to reject or accept the ”
The Power of Bias in Economics Research (2017, methodology): “Among 14,886 meta-analyses in the Cochrane Database of Systematic Reviews of medical research, Turner et al. (2013) find that 86% do not contain a single study with adequate power (≥80”
Trial Sequential Analysis in systematic reviews with meta-analysis (2017, background): “Only 22% of the metaanalyses in The Cochrane Library have 80% power to conclude whether there is an intervention effect of 30% or not when the usual maximal risks of type I error (α) of 5% and type II error (β) of 20% are applied [1].”
Power analysis for random-effects meta-analysis (2017, methodology): “We will compare the power of the individual studies and the power of the meta-analysis to which they contribute, because both types of power are important considerations, which impact on each other in practice.(5,6) The planning of future studies may be based on the power or precision of meta-analys”