Investigating Variation in Replicability: A “Many Labs” Replication Project

By R. A. Klein, K. Ratliff, M. Vianello, Reginald B. Adams, Š. Bahník, Michael J. Bernstein, K. Bocian, Mark J. Brandt, B. Brooks, C. Brumbaugh and 41 more. 2022

1,015 citations recorded by the source corpus.

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or U.S. versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

Subfields: Applied Psychology, Experimental and Cognitive Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/d0383326b54dbd083ecf88e6450ba73b46d3084e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Experimental and Cognitive Psychology

In Sociology and Political Science

Citations in the corpus

Large-Scale Replication Projects in Contemporary Psychological Research (2017, background): “Responding to this suggestion, Kennedy and Gelman (2018) propose that a Constraints on Generality statement provides the necessary information to allow for inference, via hierarchical modeling and poststratification (Park, Gelman, and Bafumi 2004), regarding how quantities of interest might vary…”
Adjusting for Publication Bias in Meta-Analysis (2016, methodology): “Further, 40% of the total variability on average across these effects was due to heterogeneity (see the I2 statistics reported in Table 3 of Klein et al., 2014).”
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants (2015, methodology): “1, prior research has suggested that a large effect (ϕ = .50) is a conservative estimate for the difference in IMC pass rates between MTurkers and online college students (Klein et al., 2014).”
The Generalizability of Survey Experiments* (2015, background): “McDermott (2002, 334) notes that performed on a convenience sample versus on a population-based sample?”
When and why defaults influence decisions: a meta-analysis of default effects (2019, result): “This result is consistent with other analyses that find that the heterogeneity of effect sizes tends to increase as the effect size increases (Klein et al., 2014).”
Using Anchor-Based Methods to Determine the Smallest Effect Size of Interest (2019, background): “…of ‘statistically significant’ as ‘meaningful’, which is becoming increasingly important given the rise of big data and the uptake of large-scale collaborative projects (e.g., Klein et al., 2014, 2018; Moshontz et al., 2018), where trivially small differences can be statistically significant.”
Estimating the reproducibility of psychological science (2015, background): “A complementary narrow-anddeep approach is characterized by the Many Labs replication projects (34).”
A Multilab Preregistered Replication of the Ego-Depletion Effect (2016, background): “Previous multilab registered replication reports also observed substantial heterogeneity in some, but not all, of the meta-analyses of the replicated effects across labs (Eerland et al., 2016; Klein et al., 2014).”
Revisiting the Red Effect on Attractiveness and Sexual Receptivity: No Effect of the Color Red on Human Mate Preferences (2016, neutral): “Indeed, it would be in line with The ‘‘Many Labs’’ Replication Project (Klein et al., 2016), when significantly stronger effect sizes are found for the original research group (i.e., the investigators who first reported the red effect (Elliot & Niesta, 2008), than publications by independent scholar”
p-Curve and Effect Size (2014, methodology): “The first relies on data from the “Many-Labs replication project” (Klein et al., 2014), in which 36 different labs around the world collaborated to run the exact same set of studies and reported all results regardless of statistical significance.”
Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research (2017, background): “The first stage consisted of several questionnaires and experimental tasks adopted fromprominent studies in psychology, which were used to assess data quality (adopted from Klein et al., 2014).”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, background): “This study and other meta-scientific investigations into the robustness of research methodologies and results (Banks et al., 2016; Bedeian et al., 2010; Begley & Ellis, 2012; Bergh et al., 2017; Camerer et al., 2016, 2018; Chang & Li, in press; Ebersole et al., 2016; Klein et al., 2014; 2018; Landy ”