Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

By Dénes Szűcs, John P. A. Ioannidis. bioRxiv, 2016

595 citations recorded by the source corpus.

We have empirically assessed the distribution of published effect sizes and estimated power by extracting more than 100,000 statistical records from about 10,000 cognitive neuroscience and psychology papers published during the past 5 years. The reported median effect size was d=0.93 (inter-quartile range: 0.64-1.46) for nominally statistically significant results and d=0.24 (0.11-0.42) for non-significant results. Median power to detect small, medium and large effects was 0.12, 0.44 and 0.73, reflecting no improvement through the past half-century. Power was lowest for cognitive neuroscience journals. 14% of papers reported some statistically significant results, although the respective F statistic and degrees of freedom proved that these were non-significant; p value errors positively correlated with journal impact factors. False report probability is likely to exceed 50% for the whole literature. In light of our findings the recently reported low replication success in psychology is realistic and worse performance may be expected for cognitive neuroscience.

Subfields: Cognitive Neuroscience, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/1fe38a6a3cfa3349ea69dcc03a9d0e05f3b0bf25

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Statistics and Probability

Citations in the corpus

Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence (2019, background): “Moreover, meta-analyses can alleviate the problem of insufficient power in many individual studies (Szucs & Ioannidis, 2017) to identify small effects that are typically encountered in psychological research (Bosco, Aguinis, Singh, Field, & Pierce, 2015; Gignac & Szodorai, 2016).”
Cross‐validation failure: Small sample sizes lead to large error bars (2017, background): “Analysis of the literature suggests that publications are indeed too often on the edge of significance (Szucs and Ioannidis, 2016) and are vastly biased by selection according to the prevailing opinions (Ioannidis, 2008).”
Psychological measurement and the replication crisis: Four sacred cows. (2020, background): “In one analysis of cognitive neuroscience research, the mean statistical power to detect small effects was only .14, meaning that many null results in this literature may be false negatives (Szucs & Ioannidis, 2017).”
When and why defaults influence decisions: a meta-analysis of default effects (2019, background): “We specifically code for decade to determine whether default effect sizes have changed over the time that they have been studied, as effect sizes in published research frequently decrease over time (Szucs et al., 2015).”
Statistical Rituals: The Replication Delusion and How We Got There (2018, background): “…neuroscience and psychology articles published between 2011 and 2014 found that the median power to detect small, medium, and large effects was 12%, 44%, and 73%, respectively; in other words, there had been no improvement since the first power studies were conducted (Szucs & Ioannidis, 2017).”
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R (2020, methodology): “Given the struggle for reproducibility in various subdomains of psychology (Ioannidis, 2005; Szucs & Ioannidis, 2017; Yong, 2012), adopting effect sizes from published data involves the risk of performing the analyses on inflated effect sizes, which in turn would result in an underpowered design.”
How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables (2019, background): “…on being true for many studies run today (Cohen, 1962; Dumas-Mallet, Button, Boraud, Gonon, & Munafò, 2017; Fraley & Vazire, 2014; Maxwell, 2004; Szucs & Ioannidis, 2017a; Vankov, Bowers, & Munafo, 2014), despite the fact that many articles on power calculation and required sample sizes have…”
With Little Power Comes Great Responsibility (2020, background): “, too few subjects or items; often taken to mean less than 80% power; Cohen, 1962) means that one is less likely to be able to draw any useful statistical conclusion from the experiment, and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis”