Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values

By Sander Greenland. American Statistician, 2019

324 citations recorded by the source corpus.

Abstract The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of “significance levels” and “P-values” in authoritative sources, and the consequent misinterpretation of P-values as error probabilities. It then discusses several properties of P-values that have been presented as fatal flaws: That P-values exhibit extreme variation across samples (and thus are “unreliable”), confound effect size with sample size, are sensitive to sample size, and depend on investigator sampling intentions. These properties are often criticized from a likelihood or Bayesian framework, yet they are exactly the properties P-values should exhibit when they are constructed and interpreted correctly within their originating framework. Other common criticisms are that P-values force users to focus on irrelevant hypotheses and overstate evidence against those hypotheses. These problems are not however properties of P-values but are faults of researchers who focus on null hypotheses and overstate evidence based on misperceptions that p = 0.05 represents enough evidence to reject hypotheses. Those problems are easily seen without use of Bayesian concepts by translating the observed P-value p into the Shannon information (S-value or surprisal) –log2(p).

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/903a16186c4d06e78e38d691e429deafe843cc61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Enhancing Rigor in Quantitative Entrepreneurship Research (2019, background): “P values continue to have value but should not be misused (Greenland, 2019; Krueger & Heck, 2019; Wasserstein & Lazar, 2016).”
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication (2018, methodology): “We do not suggest to completely abandon inference from our data to a larger population (although the title of a preprint of this paper was “Abandon statistical inference”; Amrhein, Trafimow, and Greenland 2018).”
Velocity-Based Training: From Theory to Application (2020, methodology): “This provides the practitioner a plausible range of values that are compatible with the data assumptions (34) (Figure 4, see Appendix 2, Supplemental Digital Content 2, http://links.”
There are natural scores: Full comment on Shafer,"Testing by betting: A strategy for statistical and scientific communication" (2021, background): “The one I have seen repeatedly in information statistics and data mining is the surprisal, logworth or S-value sb = logb(1/p) = −logb(p), where the log base b determines the scale via sb = se/ln(b) (Fisher 1948; Good 1956, 1957; Bayarri and Berger 1999; Boos and Stefanski 2011; Greenland 2019; Fraun”
Moving to a World Beyond “p < 0.05” (2019, unknown): “Thoughtful researchers “use a toolbox of statistical techniques, employ good judgment, and keep an eye on developments in statistical and data science,” conclude Heck and Krueger (2019), who demonstrate how the p -value can be useful to researchers as a heuristic.”