The Quarries of Meroe, Sudan: 2. Introduction

By David Colquhoun. Royal Society Open Science, 2014

569 citations recorded by the source corpus.

If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time. If, as is often the case, experiments are underpowered, you will be wrong most of the time. This conclusion is demonstrated from several points of view. First, tree diagrams which show the close analogy with the screening test problem. Similar conclusions are drawn by repeated simulations of t-tests. These mimic what is done in real life, which makes the results more persuasive. The simulation method is used also to evaluate the extent to which effect sizes are over-estimated, especially in underpowered experiments. A script is supplied to allow the reader to do simulations themselves, with numbers appropriate for their own work. It is concluded that if you wish to keep your false discovery rate below 5%, you need to use a three-sigma rule, or to insist on p≤0.001. And never use the word ‘significant’.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/edbdd4d8c8698d538836bb5abffd37b432033d05

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support (2020, unknown): “In statistics, the p-test can be misused to overestimate the probability of having found a true effect (Colquhoun, 2014; Stark and Saltelli, 2018).”
How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables (2019, background): “05) it can be shown that statistically ‘significant’ findings are more likely to be false negatives than genuine findings, certainly when they are based on underpowered studies (Colquhoun, 2014).”
Statistical Rituals: The Replication Delusion and How We Got There (2018, background): “Specifically, the blame has been laid on the “publish or perish” culture, which values quantity over quality, and on the practice of evaluating researchers by impact measures such as h-index and, more recently, “altmetrics” such as the number of tweets and mentions in blogs (Colquhoun, 2014).”
Formidable females and the power trajectories of socially integrated male vervet monkeys (2017, background): “We follow Colquhoun (2014) in describing outcomes as indicating weak ( P ~ 0.05), moderate ( P ~ 0.01) or strong ( P ~ 0.001) evidence for effects.”
Fair Statistical Communication in HCI (2016, unknown): “But Neyman–Pearson’s Type I error rate captures neither of these, even remotely (Pollard and Richardson, 1987; Colquhoun, 2014).”
Reconnecting p-Value and Posterior Probability Under One- and Two-Sided Tests (2018, methodology): “, Basic and Applied Social Psychology and Political Analysis, have made claims to ban the use of p-value in their publications (Trafimow and Marks, 2015; Gill, 2018). The controversy over p-value has recently been reignited, which is more centered around the proposals to adjust, abandon or provide a”
Experimental design and analysis and their reporting: new guidance for publication in BJP (2015, background): “…investigators have suggested that using ‘ P ’ to denote ‘statistical signiﬁcance’ as a way to denote detection of an ‘effect’ is inappropriate, and offer other solutions such as provision of effect size estimates and their precision from conﬁdence intervals (Colquhoun, 2014; Halsey et al ., 2015).”
Balancing Type I Error and Power in Linear Mixed Models (2015, background): “Aiming for high power is important for independent reasons, because even a statistically significant result obtained with a low-power experiment could easily have the wrong sign or an exaggerated effect size (Gelman & Carlin, 2014; Colquhoun, 2014).”
Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices (2017, background): “This is why, in the theory of experimental design (Box et al., 2005) factors are moved in groups, rather than OAT, to optimize the exploration of the space of the factors.”
Coup de Grâce for a Tough Old Bull: “Statistically Significant” Expires (2019, background): “(Colquhoun 2014) Many more recent papers have been published that strongly support these authors and the neoFisherian paradigm generally and warn of other problems in interpreting statistical analyses.”