The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases

By Thomas Schäfer, Marcus A. Schwarz. Frontiers in Psychology, 2019

615 citations recorded by the source corpus.

Effect sizes are the currency of psychological research. They quantify the results of a study to answer the research question and are used to calculate statistical power. The interpretation of effect sizes—when is an effect small, medium, or large?—has been guided by the recommendations Jacob Cohen gave in his pioneering writings starting in 1962: Either compare an effect with the effects found in past research or use certain conventional benchmarks. The present analysis shows that neither of these recommendations is currently applicable. From past publications without pre-registration, 900 effects were randomly drawn and compared with 93 effects from publications with pre-registration, revealing a large difference: Effects from the former (median r = 0.36) were much larger than effects from the latter (median r = 0.16). That is, certain biases, such as publication bias or questionable research practices, have caused a dramatic inflation in published effects, making it difficult to compare an actual effect with the real population effects (as these are unknown). In addition, there were very large differences in the mean effects between psychological sub-disciplines and between different study designs, making it impossible to apply any global benchmarks. Many more pre-registered studies are needed in the future to derive a reliable picture of real population effects.

Subfields: Applied Psychology, General Decision Sciences, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/4d2ba0a9eb94cd26777a42206ba6ee89530c7626

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In General Decision Sciences

In Statistics and Probability

Citations in the corpus

An Agenda for Open Science in Communication (2020, result): “…effect already exists—for example, a comparison of the effect sizes from 993 studies in psychology found that preregistered studies reported effects ( r ¼ .16) that were less than half as large compared to those reported by studies that were not preregistered (r ¼ .36; Scha¨fer & Schwarz, 2019).”
Evaluation of a decided sample size in machine learning applications (2023, methodology): “Previous studies reported that effect size is used to calculate statistical power, such as a large effect size increases power while small effect size decrease power [24, 25].”
A Meta-Analytic Review of Preschool Social and Emotional Learning Interventions (2020, methodology): “However, we know that effect size estimates within a discipline may be inflated due to publication bias, a concern that is discussed further in the following section (see Schafer & Schwarz, 2019).”
Interpreting Effect Sizes of Education Interventions (2020, methodology): “First, I update prior reviews (Bloom et al., 2008; Coe, 2002; Lipsey et al., 2012) with insights from a number of new articles (e.g., Baird & Pane, 2019; Cheung & Slavin, 2016; Funder & Ozer, 2019; Lortie-Forgues & Inglis, 2019; Schäfer & Schwarz, 2019; Simpson, 2017; Soland & Thum, 2019).”
Effect Size Guidelines, Sample Size Calculations, and Statistical Power in Gerontology (2019, background): “In addition, gerontology is a broad field, and there is doubtless variation of effect size distributions within the field, due to factors such as specific research area, specific measures used, and populations of interest (Cohen, 1962; Schäfer & Schwarz, 2019).”