PUBLICATION BIAS: THE ACHILLES’ HEEL OF SYSTEMATIC REVIEWS?

By C. Torgerson. 2006

171 citations recorded by the source corpus.

ABSTRACT: The term ‘publication bias’ usually refers to the tendency for a greater proportion of statistically significant positive results of experiments to be published and, conversely, a greater proportion of statistically significant negative or null results not to be published. It is widely accepted in the fields of healthcare and psychological research to be a major threat to the validity of systematic reviews and meta-analyses. Some methodological work has previously been undertaken, by the author and others, in the field of educational research to investigate the extent of the problem. This paper describes the problem of publication bias with reference to its history in a number of fields, with special reference to the area of educational research. Informal methods for detecting publication bias in systematic reviews and meta-analyses of controlled trials are outlined and retrospective and prospective methods for dealing with the problem are suggested.

Subfields: Management Science and Operations Research, Sociology and Political Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/44e508035b1c1b8ed1a0174d96d42e92272846b9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

What Five Decades of Research Tells Us About the Effects of Youth Psychological Therapy: A Multilevel Meta-Analysis and Implications for Science and Practice (2017, methodology): “First, we used a funnel plot (Torg- erson, 2006), with standard error plotted on the vertical axis as a function of ES on the horizontal axis.”
Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research (2015, background): “We need to be wary of publication bias (Torgerson, 2006) and encourage publication of those cases where the flipped classroom approach did not work, and be open to the possibility that flipping the classroom might not actually be the panacea that we are promised.”
Motivation Interventions in Education (2016, methodology): “The first method, the funnel plot (Light & Pillemer, 1984), is the most commonly used method to detect publication bias (Torgerson, 2006).”
Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis (2019, result): “In this way, we could evaluate the presence of publication bias in our data, which means that published studies contain a significant higher effect size than non-published studies (Torgerson, 2006).”
Avoiding Bias in Publication Bias Research: The Value of “Null” Findings (2014, background): “Given the influence of meta-analytic results on future research directions and evidence-based management, publication bias has been referred to as ‘‘the Achilles’ heel of systematic reviews’’ (Torgerson 2006), ‘‘the kryptonite of evidence-based practice’’ (Banks”
Formative assessment and feedback for learning in higher education: A systematic review (2021, background): “We also acknowledge the potential publication bias that is revealed via our review (Torgerson, 2006).”
Gender, age and tenure as moderators of work-related stressors' relationships with job performance: A meta-analysis (2008, background): “Following the recommendations of a recent review on publication bias (Torgerson, 2006) we regarded unpublished doctoral dissertations as representing unpublished studies.”
On the Effectiveness of Supplemental Instruction (2014, background): “Studies with significant, positive, large effects tend to be more attractive to journal editors than studies finding no significant effect for an educational intervention (Torgerson, 2006).”
Drivers of Performance Information Use: Systematic Literature Review and Directions for Future Research (2015, background): “The Achilles’ heel of every literature review is publication bias (Togerson, 2006).”
Human factors: spanning the gap between OM and HRM (2010, result): “the difficulty in publishing studies with negative results, may also skew the results of a literature review (Dickersin and Min, 1993; Torgerson, 2006).”
Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations (2008, background): “Publication bias, also known as the “file drawer” problem (the difficulty of finding reports of negative or null evaluations; see Cooper, 1998; Torgerson, 2006) is serious in all quantitative syntheses, but it is heightened for syntheses of program evaluations carried out by companies or their…”
Estimating the Difference Between Published and Unpublished Effect Sizes (2016, background): “Despite consistent evidence of publication bias in the medical literature, Torgerson (2006) asserted that few studies have examined its presence in the educational or psychological sciences.”
The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis (2020, background): “Publication type has been claimed to be related to the effect sizes in meta-analysis because editors, reviewers, and researchers tend to favor studies with significant effect sizes (Cornell & Mulrow, 1999; Dickersin, 2005; Torgerson, 2006).”
Meta-analysis in Second Language Research: Choices and Challenges (2010, background): “…bias, namely, the tendency of editors, reviewers, and individual researchers to favor only statistically significant or theoretically appealing findings (Moncrieff, 1998; Torgerson, 2006; Vevea & Woods, 2005; for a comprehensive review of this issue, see Rothstein, Sutton, & Borenstein, 2005).”