The Extent and Consequences of P-Hacking in Science

By M. Head, L. Holman, Robert Lanfear, A. Kahn, Michael D. Jennions. PLoS Biology, 2015

1,122 citations recorded by the source corpus.

A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as “p-hacking,” occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.

Subfields: Information Systems and Management, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a07a64ba110e0f9f7156f3bd1e376f0d2e1cddf1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems and Management

In Statistics and Probability

Citations in the corpus

Dissimilarity measures in detrital geochronology (2017, background): “This point of view is gradually gaining acceptance in many fields of science (e.g., Gelman and Stern, 2006; Head et al., 2015; Stang et al., 2010).”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, background): “Head et al. (2015) concluded that the evidence indicates the existence of “widespread” evidence for p -hacking after searching all Open Access papers in the PubMed database ( ∼ 100 , 000 papers).”
Over-optimization of academic publishing metrics: observing Goodhart’s Law in action (2018, positive): “…of the measures can be compromised, and unwelcome and unethical behaviors may develop, such as salami publications [17], ghost author-ships [18], p-hacking [19], metrics manipulation [20], faking research data [21], faking of peer reviews [22], and even plagiariz-ing by a peer reviewer [23].”
Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement (2019, neutral): “The other bias is p-hacking (Head et al. 2015).”
HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning (2019, result): “This phenomenon has been promoting the concept of SotA-hacking and publishing of marginally SotA results without in-depth analysis or discussion, similar to p-hacking , data dredging and prevalence of marginally signiﬁcant results in several other ﬁelds [15, 16, 17].”
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research* (2020, methodology): “Analysing the data using different methods and selectively reporting those with statistically significant results constitutes p-hacking and introduces bias in the literature [90,94].”
Experimental design and analysis and their reporting: new guidance for publication in BJP (2015, background): “The ﬁrst point to make is that one should not collect data then try out various statistical tests until one obtains the ‘desired’ result, known as ‘P hacking’ (Head et al ., 2015).”
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019 (2019, result): “Studies that claim to test a hypothesis but do not specify a pre-defined primary outcome measure, or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring mo”
Preferred reporting items for systematic reviews and meta‐analyses in ecology and evolutionary biology: a PRISMA extension (2021, background): “Forexample,givenmultiplechoices instatistical analyses, researchersmight favour paths leading to statistical significance (i.e. ‘P-hacking’; Simmons, Nelson & Simonsohn, 2011; Head et al., 2015).”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “If we want to prevent statistical testing problems such as p-hacking [48] to reappear in IML, we have to become more rigorous in studying and quantifying the uncertainty of IML methods.”
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020, methodology): “Analysing the data using different methods and selectively reporting those with statistically significant results constitutes p-hacking and introduces bias in the literature [90,94].”