Promises and Perils of Pre-analysis Plans

By Benjamin A. Olken. 2015

258 citations recorded by the source corpus.

The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced. I begin by laying out the basics of what a statistical analysis plan actually contains so those researchers unfamiliar with it can better understand how it is done. In so doing, I have drawn both on standards used in clinical trials, which are clearly specified by the Food and Drug Administration, as well as my own practical experience from writing these plans in economics contexts. I then lay out some of the advantages of pre-specified analysis plans, both for the scientific community as a whole and also for the researcher. I also explore some of the limitations and costs of such plans. I then review a few pieces of evidence that suggest that, in many contexts, the benefits of using pre-specified analysis plans may not be as high as one might have expected initially. For the most part, I will focus on the relatively narrow issue of pre-analysis for randomized controlled trials.

Subfields: Accounting, Sociology and Political Science, Soil Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Accounting

In Sociology and Political Science

In Soil Science

Citations in the corpus

The Econometrics of Randomized Experiments (2016, background): “Pre-analysis plans (Casey, Glennerster, and Miguel, 2012; Olken, 2015) are one approach to alleviate such concerns; another is to correct for multiple testing (List, Shaikh, and Xu, 2016).”
Split-Sample Strategies for Avoiding False Discoveries (2017, methodology): “A solution suggested by Olken (2015) suggests pre-specifying only a few primary hypotheses and using a secondary analysis, which foregos the control of false discoveries, to identify these potential surprises.”
Identification of and Correction for Publication Bias (2017, unknown): “Olken (2015). Going one step further, selective publication by journals might be eliminated by resultblind review, cf. American Society of Health Economists (2015). The Journal of Development Economics now offers authors the option of pre-results review.”
On doing relevant and rigorous experiments: Review and recommendations (2018, background): “Similarly, we have not discussed some of the limits related to field experimentation (Eden, 2017), or advanced design and statistical issues (e.g., multiple hypotheses testing corrections, List, Shaikh, & Xu, 2016); discussing pre-analysis plans would have been very useful too (e.g., Olken, 2015).”
Transparency, Reproducibility, and the Credibility of Economics Research (2018, unknown): “Olken (2015), for example, highlights both their “promises and perils.””