Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

By Jelte M. Wicherts, C. L. Veldkamp, Hilde E. M. Augusteijn, Marjan Bakker, Robbie C. M. van Aert, M. van Assen. Frontiers in Psychology, 2016

606 citations recorded by the source corpus.

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies.

Subfields: Experimental and Cognitive Psychology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/5e8902d504c03c7513f29ade98931189165a8a62

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Statistics and Probability

Citations in the corpus

Variability in the analysis of a single neuroimaging dataset by many teams (2019, background): “Participating teams were located in 17 countries/regions around the world: Australia (3 participants), Austria (3), Belgium (9), Brazil (3), Canada (16), China (2), Finland (1), France (4), Germany (21), Italy (13), the Netherlands (8), Spain (4), Sweden (4), Switzerland (3), Taiwan (3), UK (7) and ”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, background): “See Wicherts et al. 2016 for an attempt to enumerate a complete list.”
Best practices for the human evaluation of automatically generated text (2019, background): “…though each choice may be small and seemingly innocuous, collectively they can have a substantial effect on the outcome of the statistical analyses, to the extent that it becomes possible to present virtually every ﬁnding as statistically signiﬁcant (Sim-mons et al., 2011; Wicherts et al., 2016).”