Equivalence Testing for Psychological Research: A Tutorial

By Daniël Lakens, Anne M. Scheel, Peder M. Isager. Advances in Methods and Practices in Psychological Science, 2018

1,286 citations recorded by the source corpus.

Psychologists must be able to test both for the presence of an effect and for the absence of an effect. In addition to testing against zero, researchers can use the two one-sided tests (TOST) procedure to test for equivalence and reject the presence of a smallest effect size of interest (SESOI). The TOST procedure can be used to determine if an observed effect is surprisingly small, given that a true effect at least as extreme as the SESOI exists. We explain a range of approaches to determine the SESOI in psychological science and provide detailed examples of how equivalence tests should be performed and reported. Equivalence tests are an important extension of the statistical tools psychologists currently use and enable researchers to falsify predictions about the presence, and declare the absence, of meaningful effects.

Subfields: Computer Networks and Communications, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/80d5c7d24619f21bde292496735c802429910fd3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Experimental and Cognitive Psychology

Citations in the corpus

Velocity-Based Training: From Theory to Application (2020, methodology): “Two such methods that can assist this include the secondgeneration p-value (SGPV) (10,11) and tests of equivalence using 2 onesided tests (TOST) (52,53).”
Using Anchor-Based Methods to Determine the Smallest Effect Size of Interest (2019, background): “For over 60 years, researchers have pointed out the statistical benefits of specifying a range of values that are trivially small (e.g., Hodges & Lehmann, 1954; Nunnaly, 1960), and the benefits of being able to falsify predictions using equivalence tests (Rogers et al., 1993, Lakens et al., 2018).”
Requiem for a Nudge: Framing effects in nudging honesty (2020, positive): “Second, additional checks employing a Bayesian analysis and the TOST procedure ( Lakens et al., 2018 ) revealed that the null hypothesis is substantially more likely than the alternative hypothesis suggesting a difference in dishonesty between treatments.”
Social Media Use and Adolescents’ Well-Being: Developing a Typology of Person-Specific Effect Patterns (2020, positive): “Following the guidelines of Adachi and Willoughby (2015) for interpreting effects in longitudinal autoregressive models, we considered an effect size of β = .05 as the smallest effect size of interest (SESOI; Lakens et al., 2018).”
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports (2020, methodology): “…addition to testing if there is a statistically significant difference between RRs and conventional reports, we will test if the difference is smaller than our smallest effect size of interest using an equivalence test for proportion tests with an alpha level of 5% (Lakens, Scheel, & Isager, 2018).”
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R (2020, methodology): “Until then, guidance can come from previous research, literature, or practical constraints (for a more detailed discussion see Lakens, Scheel, & Isager, 2018).”
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication (2018, background): “…a measure of refutational evidence, such as a traditional P -value in a continuous fashion (as recommended by classic texts on testing such as Lehmann 1986, p. 71), reporting an observed P -value as a measure of the degree of compatibility between the hypothesis or model it tests and the…”
Sample Size Justification (2021, neutral): “If the effect size is based on a smallest effect size of interest, this value should not just be stated, but justified (e.g., based on theoretical predictions or practical implications, see Lakens, Scheel, and Isager (2018)).”