Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations

By Sander Greenland, S. Senn, Kenneth J. Rothman, J. Carlin, Charles Poole, Steven N. Goodman, Douglas G. Altman. European Journal of Epidemiology, 2016

2,814 citations recorded by the source corpus.

Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of these statistics requires an attention to detail which seems to tax the patience of working scientists. This high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so—and yet these misinterpretations dominate much of the scientific literature. In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than typically found in traditional introductory expositions. Our goal is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated analysis protocols (such as selecting analyses for presentation based on the P values they produce) can lead to small P values even if the declared test hypothesis is correct, and can lead to large P values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P values, confidence intervals, and power. We conclude with guidelines for improving statistical interpretation and reporting.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/238a09dc7231cb2524a1f165335ccd39071f5b3b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

If your P value looks too good to be true, it probably is: Communicating reproducibility and variability in cell biology (2019, background): “This paper is not an overview of statistical analyses; many have described the troubles with P values (Gardner and Altman, 1986; Sullivan and Feinn, 2012; Greenland et al., 2016) and there are several”
Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values (2019, background): “Confusion of the observed P-value p with the random variable P may be a major contributor to some of the fallacies described below and earlier (e.g., Greenland et al. 2016).”
Editorial: Enhancing quantitative theory-testing entrepreneurship research (2019, background): “…of a finding using the p-value is the assumption that the null hypothesis is true; the p-value simply being the probability that the observed effect or a larger effect is due to random variation under the assumption of a true null hypothesis (Greenland et al., 2016; Wasserstein and Lazar, 2016).”
For and Against Methodologies: Some Perspectives on Recent Causal and Statistical Inference Debates (2017, methodology): “For example, saying the P-value measures ‘‘the probability that the difference is due to sampling error’’ or ‘‘chance’’ is logically no different than claiming the P-value is the probability of the null hypothesis, which it is emphatically not (Wasserstein and Lazar 2016; Greenland et al. 2016).”
The statistical significance filter leads to overoptimistic expectations of replicability (2018, background): “Many researchers have pointed out that we should aim for higherprecision estimates and focus on estimation rather than only focusing on statistical significance (e.g., Claridge-Chang & Assam, 2016; Greenland et al., 2016).”
Enhancing Rigor in Quantitative Entrepreneurship Research (2019, methodology): “…recommend that authors not rely on seemingly random cutoff values to judge the practical significance of their findings but also incorporate qualitative judgments that consider what really matters to practitioners in the specific context of the study (Aguinis et al., 2010; Greenland et al., 2016).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “not significant) and common misinterpretations [33, 35, 73] such as 1) lack of statistically significant results does not demonstrate the absence of effect (Figure 2, right), and 2) given enough data, any trivial effect can be statistically significant but may not be practically significant.”
An Introduction to Second-Generation p-Values (2019, background): “In practice, these flaws often overshadow the p-value’s positive attributes (Berger and Sellke 1987; Blume and Peipert 2003; Greenland et al. 2016; Goodman 1993; Cornfield 1966; Royall 1986; Dupont 1983).”
A Tutorial On the design, experimentation and application of metaheuristic algorithms to real-World optimization problems (2021, background): “This same problem could also arise when using conﬁdence intervals methods, but it has been proven that it happens in a smaller scale [107].”
Coup de Grâce for a Tough Old Bull: “Statistically Significant” Expires (2019, background): “Fortunately, a recent authoritative, comprehensive and detailed review by Greenland et al. (2016) makes it easy for readers to catch up with the best thinking on these topics.”
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication (2018, background): “For example, with a statistical power of 80%, two studies will be traditionally judged as “conflicting,” meaning that one is “significant” and the other is not, in one third of the cases (Greenland et al. 2016; Amrhein, Korner-Nievergelt, and Roth 2017).”
Peaks Over Threshold (POT): A methodology for automatic threshold estimation using goodness of fit p‐value (2017, methodology): “…can be viewed as a continuous measure of the compatibility between the data and the entire model used to compute it, ranging from 0 for complete incompatibility to 1 for perfect compatibility, and in this sense may be viewed as measuring the fit of the model to the data’’ [Greenland et al., 2016].”