Using Bayes to get the most out of non-significant results

By Zoltan Dienes. Frontiers in Psychology, 2014

1,936 citations recorded by the source corpus.

No scientific conclusion follows automatically from a statistically non-significant result, yet people routinely use non-significant results to guide conclusions about the status of theories (or the effectiveness of practices). To know whether a non-significant result counts against a theory, or if it just indicates data insensitivity, researchers must use one of: power, intervals (such as confidence or credibility intervals), or else an indicator of the relative evidence for one theory over another, such as a Bayes factor. I argue Bayes factors allow theory to be linked to data in a way that overcomes the weaknesses of the other approaches. Specifically, Bayes factors use the data themselves to determine their sensitivity in distinguishing theories (unlike power), and they make use of those aspects of a theory’s predictions that are often easiest to specify (unlike power and intervals, which require specifying the minimal interesting value in order to address theory). Bayes factors provide a coherent approach to determining whether non-significant results support a null hypothesis over a theory, or whether the data are just insensitive. They allow accepting and rejecting the null hypothesis to be put on an equal footing. Concrete examples are provided to indicate the range of application of a simple online Bayes calculator, which reveal both the strengths and weaknesses of Bayes factors.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/e898185ae1b4f09425720ec69f79cdceffc98200

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore, SJR, SNIP, JCR Impact Factors (2019, background): “To obtain evidence against a hypothesis, the researcher needs to calculate the power of the test and then the effect size to demonstrate that there is no ‘non-negligible’ effect (see Cohen et al., 2003; Lakens, 2013), or alternatively use Bayes factors (see Dienes, 2014).”
Indices of Effect Existence and Significance in the Bayesian Framework (2019, background): “, they allow to quantify the evidence in favor of the null hypothesis (Dienes, 2014; Jarosz and Wiley, 2014).”
J. B. S. Haldane's Contribution to the Bayes Factor Hypothesis Test (2015, methodology): “, 2014), psychology (Dienes, 2014), economics (Malikov et al.”
Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP (2018, background): “As this BF01 value was close to 1, this is suggestive of data insensitivity [4].”
Equivalence Tests (2017, neutral): “Bayesian methods can also be used to test a null effect (e.g., Dienes, 2014), but equivalence tests do not require researchers to switch between statistical philosophies to test the absence of a meaningful effect, and the availability of power analyses for equivalence tests allows researchers to…”
Bayesian Analysis of Factorial Designs (2017, background): “We tend to think that the prior should reflect a reasonable set of beliefs informed by context and the literature (Dienes, 2014).”
How can we measure awareness? An overview of current methods (2015, background): “…of subliminal perception can be attributed to a failure to recognize the limitations of the dissociation logic, compounded by the inherent statistical limitations in reasoning based on null effects (for a discussion of how Bayesian approaches may help address this latter challenge see Dienes 2014).”
Fair Statistical Communication in HCI (2016, background): “Despite this, few researchers can resist the temptation to conclude that there is no effect, a common fallacy called “accepting the null” which had frequently led to misleading or wrong scientific conclusions (Dienes, 2014, p.1).”
The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective (2017, neutral): “This is highly desirable for theoretical domains in which “proving” the null is the goal (e.g., Dienes, 2014, 2016; Gallistel, 2009; Lee & Wagenmakers, 2014; Rouder et al., 2009; Wagenmakers, 2007).”
Bayes factor design analysis: Planning for compelling evidence (2016, background): “Note that all computations and procedures outlined here are not restricted to these specific choices and can be easily generalized to undirected tests and all other flavors of Bayes factors as well (Dienes, 2014).”
How Do I Know What My Theory Predicts? (2019, methodology): “None of these models may be appropriate in any given case (e.g., see Dienes, 2014; Gronau, Ly, & Wagenmakers, 2019); however, they are good enough approximations sufficiently often that they serve as good vehicles for discussing the heuristics this article focuses on (see Dienes & McLatchie, 2018,…”
How Bayes factors change scientific practice (2016, methodology): “One way of distinguishing H1 from H0 is by use of inference by intervals (Dienes, 2014).”
How Bayesian statistics are needed to determine whether mental states are unconscious (2015, neutral): “If a significant result at the 5% level is obtained and the obtained effect size is about that expected, then B is likely to be about 3 (Dienes, 2014).”
Sample Size Justification (2021, neutral): “For this reason an a-priori power based on a smallest effect size of interest is generally preferred, whenever researchers are able to specify one (Aberson, 2019; Albers & Lak-ens, 2018; Brown, 1983; Cascio & Zedeck, 1983; Dienes, 2014; Lenth, 2001).”