Abandon Statistical Significance

By Blakeley B. McShane, David Gal, Andrew Gelman, Christian P. Robert, J. Tackett. American Statistician, 2017

860 citations recorded by the source corpus.

ABSTRACT We discuss problems the null hypothesis significance testing (NHST) paradigm poses for replication and more broadly in the biomedical and social sciences as well as how these problems remain unresolved by proposals involving modified p-value thresholds, confidence intervals, and Bayes factors. We then discuss our own proposal, which is to abandon statistical significance. We recommend dropping the NHST paradigm—and the p-value thresholds intrinsic to it—as the default statistical paradigm for research, publication, and discovery in the biomedical and social sciences. Specifically, we propose that the p-value be demoted from its threshold screening role and instead, treated continuously, be considered along with currently subordinate factors (e.g., related prior evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain) as just one among many pieces of evidence. We have no desire to “ban” p-values or other purely statistical measures. Rather, we believe that such measures should not be thresholded and that, thresholded or not, they should not take priority over the currently subordinate factors. We also argue that it seldom makes sense to calibrate evidence as a function of p-values or other purely statistical measures. We offer recommendations for how our proposal can be implemented in the scientific publication process as well as in statistical decision making more broadly.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/26e697fc7ecd3592a6e120cc8cae48d76b01f93d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

The significance filter, the winner's curse and the need to shrink (2020, background): “, 2018) also reduces power, and therefore may backfire by aggravating the winner’s curse (McShane et al., 2019).”
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018, background): “Some papers are even talking about abandoning the null hypothesis statistical significance test approach due to this hard-to-meet assumption (Koplenig, 2017; McShane et al., 2017; Carver, 1978; Leek et al., 2017).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “not significant) and common misinterpretations [33, 35, 73] such as 1) lack of statistically significant results does not demonstrate the absence of effect (Figure 2, right), and 2) given enough data, any trivial effect can be statistically significant but may not be practically significant.”
Safe Testing (2019, background): “Cumming (2012)) has been advocating for generally replacing testing by estimation accompanied by confidence intervals; or, more generally (McShane et al., 2019),”
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, background): “Note that some argue against statistical significance tests for NLP because the independence assumptions do not necessarily hold (Koplenig, 2019; McShane et al., 2019; Carver, 1978; Leek et al., 2017).”
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication (2018, background): “Such model assumptions are part of what are called “auxiliary hypotheses” (Popper 1968), “auxil-iary theories” (Meehl 1990), or “currently subordinate factors” (McShane et al. 2019).”
Three Recommendations for Improving the Use of p-Values (2019, background): “Such dichotomization is often harmful because it obscures the uncertainty that is intrinsic to any statistical evidence (e.g., Amrhein and Greenland 2018; McShane and Gelman 2017).”
An imprecise-probabilistic characterization of frequentist statistical inference (2021, background): “Pushes to ban p-values (Trafimow and Marks 2015) or to abandon statistical significance (McShane et al. 2019) are aimed at preventing “p < 0.”
Three ways to make replication mainstream (2017, methodology): “ting dichotomous declarations of truth or falsity, binary statements about there being “an effect” or “no effect,” a “successful replication” or a “failed replication”, and (iii) should be abandoned (McShane et al., 2017; McShane and Gelman, 2017). Instead, we would like to see replication efforts s”
With Little Power Comes Great Responsibility (2020, background): “NHST is not free from problems, in that certain systematic processes within the practice of scientific research and publishing can undermine its advantages, many of which have been explored in the literature (Gelman and Loken, 2013; Ioannidis, 2019; McShane et al., 2019).”
Moving to a World Beyond “p < 0.05” (2019, neutral): “” Referring to replication studies in psychology, McShane et al. (2019) recommend that future large-scale replication projects “should follow the ‘one phenomenon, many studies’ approach of the Many Labs project and Registered Replication Reports rather than the ‘many phenomena, one study’ approach…”
Sample size evolution in neuroimaging research: an evaluation of highly-cited studies (1990-2012) and of latest practices (2017-2018) in high-impact journals (2019, background): “Further, increasing the α level will also greatly increase the amount of false positive results if the null hypothesis is true (for discussions regarding the α level see Benjamin et al., 2018 and Lakens et al., 2018; Wasserstein et al., 2019; Amrhein et al., 2019; McShane et al., 2019).”
Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values (2019, background): “While there are ongoing calls to abolish misleading jargon involving “significance” (Amrhein et al. 2018; McShane et al. 2018), almost no attempt has been made to correct Fisher’s mistake of using “null hypothesis” for any tested hypothesis H, ignoring that in ordinary English “null” is a synonym…”
Enhancing Rigor in Quantitative Entrepreneurship Research (2019, background): “prevailing null hypothesis significance testing (NHST) paradigm (e.g., Gigerenzer, 2004; Matthews, 2019; McShane et al., 2019; Nosek et al., 2015; Schwab et al., 2011; Wasserstein et al., 2019),1 rapid developments in empirical methods and software tools also create new opportunities for…”
Best practices for the human evaluation of automatically generated text (2019, background): “In view of these issues, some have argued that NHST should be abandoned (Koplenig, 2017; McShane et al., 2019).”
Reconnecting p-Value and Posterior Probability Under One- and Two-Sided Tests (2018, background): “McShane et al. (2019) recommended that the role of p-value as a threshold for screening scientific findings should be demoted, and that pvalue should not take priority over other statistical measures.”
Estimating the Reproducibility of Experimental Philosophy (2018, background): “Thus, p-values should serve as just one piece of information out of many such pieces in a robust, flexible, and contextsensitive inferential process (American Statistical Association 2016; Lakens et al. 2017; McShane et al. 2017; Murtaugh 2014).”