The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey

By John P. A. Ioannidis, Thomas A. Trikalinos. Canadian Medical Association Journal, 2007

996 citations recorded by the source corpus.

Background: Statistical tests for funnel-plot asymmetry are common in meta-analyses. Inappropriate application can generate misleading inferences about publication bias. We aimed to measure, in a survey of meta-analyses, how frequently the application of these tests would be not meaningful or inappropriate. Methods: We evaluated all meta-analyses of binary outcomes with é 3 studies in the Cochrane Database of Systematic Reviews (2003, issue 2). A separate, restricted analysis was confined to the largest meta-analysis in each of the review articles. In each meta-analysis, we assessed whether criteria to apply asymmetry tests were met: no significant heterogeneity, I2 < 50%, é 10 studies (with statistically significant results in at least 1) and ratio of the maximal to minimal variance across studies > 4. We performed a correlation and 2 regression asymmetry tests and evaluated their concordance. Finally, we sampled 60 meta-analyses from print journals in 2005 that cited use of the standard regression test. Results: A total of 366 of 6873 (5%) and 98 of 846 meta-analyses (12%) in the wider and restricted Cochrane data set, respectively, would have qualified for use of asymmetry tests. Asymmetry test results were significant in 7%–18% of the meta-analyses. Concordance between the 3 tests was modest (estimated k 0.33–0.66). Of the 60 journal meta-analyses, 7 (12%) would qualify for asymmetry tests; all 11 claims for identification of publication bias were made in the face of large and significant heterogeneity. Interpretation: Statistical conditions for employing asymmetry tests for publication bias are absent from most meta-analyses; yet, in medical journals these tests are performed often and interpreted erroneously.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/9994a647f6007f455961b0ffbcdb4fddfb1e839d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Daily Sitting Time and All-Cause Mortality: A Meta-Analysis (2013, background): “Evidence of publication bias The small number of studies and presence of significant heterogeneity precluded formal testing for publication bias[35].”
The trim-and-fill method for publication bias: practical guidelines and recommendations based on a large database of meta-analyses (2019, methodology): “the funnel plot’s asymmetry is truly caused by publication bias or confounded by heterogeneity.([49]) More sophisticated statistical methods (e.”
The effects of vitamin D on skeletal muscle strength, muscle mass, and muscle power: a systematic review and meta-analysis of randomized controlled trials. (2014, methodology): “Moreover, because of the limited number of studies included in the meta-analyses on muscle mass and muscle power and because of the high heterogeneity observed in the metaanalysis on muscle strength, we were unable to measure the potential publication bias by the Begg’s adjusted rank correlation and”
Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials (2011, methodology): “After reviewing the results of these studies, and based on theoretical considerations, we formulated recommendations on testing for funnel plot asymmetry (box 2). The appendix describes the proposed tests, explains the reasons that some were not recommended, and discusses funnel plots for interventi”
Intensive insulin therapy and mortality among critically ill patients: a meta-analysis including NICE-SUGAR study data (2009, methodology): “bias in our analysis, the tests we used may be unreliable in the presence of significant heterogeneity.(41) However, since neither peer reviewers nor we have been able to assess the methodologic quality of the unpublished trials, we decided to include only trials published either in full or as abstr”