Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

By K. Dwan, Douglas G. Altman, J. -. Arnaiz, J. Bloom, A. Chan, E. Cronin, E. Decullier, P. Easterbrook, E. von Elm, C. Gamble and 4 more. PLoS ONE, 2008

1,504 citations recorded by the source corpus.

Background The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Subfields: Sociology and Political Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/bce6828769841b4ca45147b876d9a0aef8ebf612

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials (2010, background): “Empirical evidence suggests that authors often ignore, in the conduct and reporting of their trial, what they stated in the protocol (4, 22).”
CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. (2012, background): “A review of comparisons with protocols showed that about half of journal articles describing RCTs had an unexplained discrepancy in the primary outcomes [57].”
The Extent and Consequences of P-Hacking in Science (2015, methodology): “Although methods and results in publications do not always match their prespecified protocols [5,66], prespecification allows readers to assess the risk of p-hacking and adjust their confidence in the reported outcomes accordingly.”
Deep impact: unintended consequences of journal rank (2013, background): “This is the well-known phenomenon of publication bias (Song et al., 1999; Van Dongen, 2011; Munafò et al., 2007; Young et al., 2008; Callaham, 2002; Møller and Jennions, 2001; Møller et al., 2005; Schooler, 2011; Dwan et al., 2008).”
The Chrysalis Effect (2017, neutral): “…and protocols against published journal articles for evidence of engagement in QRPs (e.g., Chan, Hrobjartsson, Jorgensen, Gotzsche, & Altman, 2008; Dwan et al., 2008, 2011; Huic, Marusic, & Marusic, 2011; Mathieu, Boutron, Moher, Altman, & Ravaud, 2009; Turner, Knoepflmacher, & Shapley, 2012).”
A Call for Examining Replication and Bias in Special Education Research (2014, background): “Dwan et al. (2008) reviewed studies that compared research protocols (registered prior to the start of studies) to subsequent research reports. They reported between 13% and 31% of primary outcomes specified in protocols were omitted in publications. Moreover, statistically significant outcomes were”
Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. (2010, background): “Sample size, median (IQR) [range] 84 (46-206) [4-6848]”
Do Persuasive Technologies Persuade? - A Review of Empirical Studies (2014, background): “For example, one analysis, looking mainly at studies in the medical field [18], has indicated that publication and outcome reporting biases are prevalent and affect the published research.”
Korean translation of the CONSORT 2010 Statement: updated guidelines for reporting parallel group randomized trials (2014, neutral): “저자들은 때로 시험의 수행과 보고에서 있어 연구계획서에 기술했던 내용을 무시한다는 경험적 증거들이 있다 [4,22].”
Publication bias in the social sciences: Unlocking the file drawer (2014, methodology): “A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”