The preregistration revolution

By Brian A. Nosek, C. Ebersole, A. DeHaven, D. Mellor. Proceedings of the National Academy of Sciences of the United States of America, 2018

1,563 citations recorded by the source corpus.

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

View Paper: https://www.semanticscholar.org/paper/5465884a4aad5dc256ce5b4bfacdfa7cdc2c78ee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A practical guide to question formation, systematic searching and study screening for literature reviews in ecology and evolution (2021, background): “As part of the credibility revolution, there is a growing emphasis on registering one's study (Nosek et al., 2018; Parker et al., 2019).”
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019 (2019, background): “Other online resources include the Open Science Framework [224], which is suitable to deposit PHISPS (Population; Hypothesis; Intervention; Statistical Analysis Plan; Primary; Outcome Measure; Sample Size Calculation) protocols [225] and provide researchers with the flexibility to embargo the prereg”
Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research* (2019, methodology): “To start, preregistration involves having one’s research design, hypotheses, and analysis plan posted and date-stamped prior to data collection and analysis so that one does not confuse prediction with postdiction: post hoc explanations (Nosek et al., 2018) or ‘‘HARKing’’ (hypothesizing after result”
Estimating the Reproducibility of Experimental Philosophy (2018, background): “Meanwhile, promising ideas for improvement—including the recent push toward norms of pre-registration— are now gaining traction among leading scientists (Chambers & Munafò, 2013; Munafò et al., 2017; Nosek et al., 2018; but see Lash & Vandenbroucke, 2012; Scott, 2013).”
Variability in the analysis of a single neuroimaging dataset by many teams (2019, unknown): “18 (9), Brazil (3), Canada (16), China (2), Finland (1), France (4), Germany (21), Italy (13), the Netherlands (8), Spain (4), Sweden (4), Switzerland (3), Taiwan (3), UK (7) and the USA (76).”
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate (2017, background): “In summary, several researchers have argued that p values lose their meaning in exploratory analyses (Bender & Lange, 2001; Dahl et al., 2008; de Groot, 1956/2014; Forstmeier et al., 2016; Nosek et al., 2017; Nosek & Lakens, 2014; Wagenmakers, 2016).”
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings (2018, methodology): “Upon completion of an exploratory analysis, the analytic scripts were registered and applied to the holdout data for a mostly confirmatory test (Nosek et al., 2018).”
An Agenda for Open Science in Communication (2020, background): “Crucially, HARKing capitalizes on chance: Unexpected results might not represent stable effects, which dilutes the literature with false positives and contributes to low replicability (Nosek et al., 2018).”