Large-Scale Replication Projects in Contemporary Psychological Research

By Blakeley B. McShane, J. Tackett, Ulf Böckenholt, Andrew Gelman. American Statistician, 2017

93 citations recorded by the source corpus.

ABSTRACT Replication is complicated in psychological research because studies of a given psychological phenomenon can never be direct or exact replications of one another, and thus effect sizes vary from one study of the phenomenon to the next—an issue of clear importance for replication. Current large-scale replication projects represent an important step forward for assessing replicability, but provide only limited information because they have thus far been designed in a manner such that heterogeneity either cannot be assessed or is intended to be eliminated. Consequently, the nontrivial degree of heterogeneity found in these projects represents a lower bound on the true degree of heterogeneity. We recommend enriching large-scale replication projects going forward by embracing heterogeneity. We argue this is the key for assessing replicability: if effect sizes are sufficiently heterogeneous—even if the sign of the effect is consistent—the phenomenon in question does not seem particularly replicable and the theory underlying it seems poorly constructed and in need of enrichment. Uncovering why and revising theory in light of it will lead to improved theory that explains heterogeneity and increases replicability. Given this, large-scale replication projects can play an important role not only in assessing replicability but also in advancing theory.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, General Decision Sciences

View Paper: https://www.semanticscholar.org/paper/8a328046ce65b5bd21d75769d8b75fcf1c5ab459

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

Moving to a World Beyond “p < 0.05” (2019, unknown): “” Referring to replication studies in psychology, McShane et al. (2019) recommend that future large-scale replication projects “should follow the ‘one phenomenon, many studies’ approach of the Many Labs project and Registered Replication Reports rather than the ‘many phenomena, one study’ approach…”
What Can Be Learned from Growth Mindset Controversies? (2020, background): “Therefore the Rienzo study is not exactly evidence against mindset effects (see a discussion of this statistical argument in McShane et al., 2019).”