Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
By Abel Brodeur, Nikolai Cook, Anthony Heyes. The American Economic Review, 2020
260 citations recorded by the source corpus.
The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time. (JEL A14, C12, C52)
Subfields: Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/f53c6450d4f877af7fb1a8c49f9d8d8957adf3ac
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Statistics and Probability
- E. Linos
- K. Muralidharan
- Mauricio Romero
- K. Wuthrich
- J. Schwarz
- Josef Bajzík
- Stefano DellaVigna
- Tomáš Havránek
Citations in the corpus
- RCTs to Scale: Comprehensive Evidence from Two Nudge Units (2020, methodology): “As a second test, following Brodeur et al. (2016), in Figure 5b we plot the distribution of t statistics around the standard 5% significant threshold (t=1.”
- Estimating the Armington elasticity: The importance of study design and publication bias (2020, background): “…Alinaghi & Reed (2018), Christensen & Miguel (2018), Doucouliagos et al. (2018), Bom & Rachinger (2019), Campos et al. (2019), Blanco-Perez & Brodeur (2020), Brodeur et al. (2020), and Ugur et al. (2020). sampling error; for the same reason, researchers will sometimes obtain estimates much…”
- Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments (2019, methodology): “Finally, we contribute to the literature on the pitfalls of focusing on statistical significance in applied work (e.g., Brodeur et al., 2016; Wasserstein & Lazar, 2016; Amrhein et al., 2019; Wasserstein et al., 2019; Brodeur et al., 2020).”