Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*

By Alwyn Young. 2018

558 citations recorded by the source corpus.

I follow R. A. Fisher'sThe Design of Experiments (1935), using randomization statistical inference to test the null hypothesis of no treatment effects in a comprehensive sample of 53 experimental papers drawn from the journals of the American Economic Association. In the average paper, randomization tests of the significance of individual treatment effects find 13% to 22% fewer significant results than are found using authors’ methods. In joint tests of multiple treatment effects appearing together in tables, randomization tests yield 33% to 49% fewer statistically significant results than conventional tests. Bootstrap and jackknife methods support and confirm the randomization results.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

The Econometrics of Randomized Experiments (2016, methodology): “See for recent discussions Rosenbaum (1992), Gail, Tian and Piantadosi (1988), and Imbens and Rubin (2015), and in economics Young (2016).”
Priming in Economics (2016, background): “While this criticism should be taken seriously, problems with replication have been observed throughout psychological research [55] and other sciences [56], including economics [57].”
Non-durable consumption and housing net worth in the Great Recession: Evidence from easily accessible data (2016, unknown): “Thewell-knownwork of Young (2019) on the robustness of estimated treatment effects in RCTs is a prominent example of this type of exercise.”
Synthetic Control Method: Inference, Sensitivity Analysis and Confidence Sets (2018, background): “…observed in contexts that are different from ours: Lehmann (1959) looks to a simple test of mean differences, Ibragimov & Muller (2010) analyzes a two-sample test of mean differences where samples’ variances are different from each other, and Young (2015) focus on a linear regression coefficient.”
Cluster-robust inference: A guide to empirical practice (2022, methodology): “Young (2019) contains an interesting recent application of RI, where the RI-β and RI-t procedures are applied to regressions for the results of 53 randomized experiments in published papers.”
Understanding and misunderstanding randomized controlled trials (2016, unknown): “In addition to Young (2017), Imbens and Kolesár (2016) provide practical advice for dealing with the Fisher-Behrens problem, and the best current practice tries to be careful about multiple hypothesis testing.”
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments (2019, methodology): “Banerjee, A., & Duflo, E. (2005). Chapter 7 growth theory through the lens of development economics. In P. Aghion & S. N. Durlauf (Eds.), (Vol. 1, p. 473 - 552). Elsevier. Bertrand, M., Karlan, D., Mullainathan, S., Shafir, E., & Zinman, J. (2010). What’s advertising content worth? evidence from a c”
Econometric Methods for Program Evaluation (2018, methodology): “This is explained in Ding (2017) using the power properties of the two types of tests, and illustrated in Young (2017).”
A Theory of Experimenters: Robustness, Randomization, and Balance (2020, unknown): “First, formally specifying the preferences of stakeholders permits randomization inference tests (Fisher 1935, Morgan and Rubin 2012, Young 2016).”