Statistical Rituals: The Replication Delusion and How We Got There

By Gerd Gigerenzer. Advances in Methods and Practices in Psychological Science, 2018

213 citations recorded by the source corpus.

The “replication crisis” has been attributed to misguided external incentives gamed by researchers (the strategic-game hypothesis). Here, I want to draw attention to a complementary internal factor, namely, researchers’ widespread faith in a statistical ritual and associated delusions (the statistical-ritual hypothesis). The “null ritual,” unknown in statistics proper, eliminates judgment precisely at points where statistical theories demand it. The crucial delusion is that the p value specifies the probability of a successful replication (i.e., 1 – p), which makes replication studies appear to be superfluous. A review of studies with 839 academic psychologists and 991 students shows that the replication delusion existed among 20% of the faculty teaching statistics in psychology, 39% of the professors and lecturers, and 66% of the students. Two further beliefs, the illusion of certainty (e.g., that statistical significance proves that an effect exists) and Bayesian wishful thinking (e.g., that the probability of the alternative hypothesis being true is 1 – p), also make successful replication appear to be certain or almost certain, respectively. In every study reviewed, the majority of researchers (56%–97%) exhibited one or more of these delusions. Psychology departments need to begin teaching statistical thinking, not rituals, and journal editors should no longer accept manuscripts that report results as “significant” or “not significant.”

Subfields: Ecology, Evolution, Behavior and Systematics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/d2545c297353871209521ecd0604667716a9ef93

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology, Evolution, Behavior and Systematics

Blakeley B. McShane

In Statistics and Probability

Citations in the corpus

Abandon Statistical Significance (2017, background): “For related results, see Oakes (1986), Cohen (1994), Haller and Krauss (2002), and Gigerenzer (2018).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “not significant) and common misinterpretations [33, 35, 73] such as 1) lack of statistically significant results does not demonstrate the absence of effect (Figure 2, right), and 2) given enough data, any trivial effect can be statistically significant but may not be practically significant.”
The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support (2020, unknown): “…’ (Saltelli and Funtowicz, 2017; Saltelli, 2018) where misuse or abuse of statistics (Stark and Saltelli, 2018; Gigerenzer and Marewski, 2015; Leek et al., 2017; Singh Chawla, 2017; Wasserstein and Lazar, 2016; Gigerenzer, 2018) is often cited as the root cause of the crisis (Ioannidis, 2005).”