Power analysis for random-effects meta-analysis

By Dan Jackson, Rebecca M. Turner. Research Synthesis Methods, 2017

549 citations recorded by the source corpus.

One of the reasons for the popularity of meta‐analysis is the notion that these analyses will possess more power to detect effects than individual studies. This is inevitably the case under a fixed‐effect model. However, the inclusion of the between‐study variance in the random‐effects model, and the need to estimate this parameter, can have unfortunate implications for this power. We develop methods for assessing the power of random‐effects meta‐analyses, and the average power of the individual studies that contribute to meta‐analyses, so that these powers can be compared. In addition to deriving new analytical results and methods, we apply our methods to 1991 meta‐analyses taken from the Cochrane Database of Systematic Reviews to retrospectively calculate their powers. We find that, in practice, 5 or more studies are needed to reasonably consistently achieve powers from random‐effects meta‐analyses that are greater than the studies that contribute to them. Not only is statistical inference under the random‐effects model challenging when there are very few studies but also less worthwhile in such cases. The assumption that meta‐analysis will result in an increase in power is challenged by our findings.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/10b27e424a08593277d90791c25e768fa7e55f3f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

How to unring the bell: A meta-analytic approach to correction of misinformation (2018, result): “In line with the finding that five or more studies are needed to reasonably achieve power from random-effects models that are greater than the statistical power of the individual studies (Jackson & Turner, 2017), moderation analyses were conducted for categories that included at least five cases.”
Research self-efficacy: A meta-analysis (2021, unknown): “(Jackson & Turner, 2017).”
Evaluating the Impact of Attempts to Correct Health Misinformation on Social Media: A Meta-Analysis (2020, methodology): “Based on recommendation by Jackson and Turner (2017), moderation tests were performed only when the data included at least five cases for each level of the moderator (see Table 1 for an overview of studies included in the meta-analysis).”
The potential of digital tools to enhance mathematics and science learning in secondary schools: A context-specific meta-analysis (2020, background): “The small number of studies in most subgroups and the use of a random-effects model resulted in reduced statistical power, which may partly explain non-significant effects and between-levels variance in the moderator analyses (Jackson & Turner, 2017).”
Fact-Checking: A Meta-Analysis of What Works and for Whom (2019, methodology): “Due to concerns over statistical power, all moderation analyses included, at least, five cases for each value (for a discussion of statistical power in meta-analysis see Jackson & Turner, 2017).”