The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

By Andrew Gelman, Hal S. Stern. 2006

1,045 citations recorded by the source corpus.

Subfields: Atomic and Molecular Physics, and Optics, Social Psychology, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/541d44f4542a241f7d42adcd723bf1b7ddab5b09

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Social Psychology

In Statistical and Nonlinear Physics

Citations in the corpus

A General Framework for Comparing Predictions and Marginal Effects across Models (2019, methodology): “…differences by examining only the statistical significance of an effect within each model; this approach has been criticized because it is necessary to directly test cross-model differences to determine their statistical significance (Gelman and Stern 2006; Mustillo, Lizardo, and McVeigh 2018).”
Moving to a World Beyond “p < 0.05” (2019, neutral): “As Gelman and Stern (2006) famously observed, the difference between “significant” and “not significant” is not itself statistically significant.”
Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values (2019, background): “It is often noted that P-values vary dramatically from sample to sample even under ideal experimental replications (Goodman 1992; Senn 2001,2002; Gelman and Stern 2006; Murdoch et al. 2008; Boos and Stefanski 2011).”
A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: the case of felt power (2017, background): “…with respect to the size of the effect of power posing on felt power, although a formal assessment of this possibility requires a different statistical analysis (e.g., Gelman & Stern, 2006; Nieuwenhuis, Forstmann, & Wagenmakers, 2011), the development of which is beyond the scope of this paper.”
The roles of information deficits and identity threat in the prevalence of misperceptions (2019, result): “…negative but 3We find no evidence of significant imbalance on covariates in balance tests (results available upon request). not significant (p < .15), though we cannot directly reject the null of no difference in Graph effects between withdrawal opponents and other groups (Gelman and Stern 2006).”
For and Against Methodologies: Some Perspectives on Recent Causal and Statistical Inference Debates (2017, background): “The medical literature abounds in parallel absurdities, such as writing as if there is a meaningful difference between P=0.04 and P=0.06 when there is not (Tukey 1962, p. 62; Gelman and Stern 2006).”
p-Curve and Effect Size (2014, neutral): “Researchers interested in publishing attenuated interactions need the interaction terms to be significant (Gelman & Stern, 2006).”
Erroneous analyses of interactions in neuroscience: a problem of significance (2011, background): “Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significan”
Displaying Uncertainty With Shading (2008, background): “The casual use of hypothesis tests based on arbitrary thresholds is frequently criticized (Gelman and Stern 2006), particularly in medical research (Sterne et al. 2001).”
I can see clearly now: Reinterpreting statistical significance (2018, background): “The difference between “clear” and “not clear” is not clear (Gelman and Stern, 2006) This correlation was significant in males (ρ = 0.”
Strengthening statistical usage in marine ecology (2012, background): “Statisticians have decried this reasoning for decades (Cohen, 1994; Fisher, 1959; Gelman and Stern, 2006; Gigerenzer, 2004; Gigerenzer et al., 2004; Hubbard and Bayarri, 2003; Stephens et al., 2005; and the numerous references cited in these works).”