Many Labs 3: Evaluating participant pool quality across the academic semester via replication

By C. Ebersole, Olivia E. Atherton, A. Belanger, Hayley M. Skulborstad, Jill Allen, J. Banks, E. Baranski, Michael J. Bernstein, D. Bonfiglio, L. Boucher and 54 more. 2016

374 citations recorded by the source corpus.

Abstract The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

Subfields: Applied Psychology, Experimental and Cognitive Psychology, Literature and Literary Theory

View Paper: https://www.semanticscholar.org/paper/dfad0bb590673bd0e06aee182baa2d091052e2ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Experimental and Cognitive Psychology

In Literature and Literary Theory

Citations in the corpus

The Replication Crisis in Psychology: An Overview for Theoretical and Philosophical Psychology (2019, methodology): “These have included the Many Labs projects (Ebersole et al., 2016; Klein et al., 2018), the Registered Replication Reports initiative sponsored by the Association for Psychological Science (Wagenmakers et al., 2016; Hagger et al., 2016; O’Donnell, et al., 2018), and the Psychological Science…”
Violating the normality assumption may be the lesser of two evils (2018, background): “In a growing body of literature, researchers express their concerns about irreproducible results (Open Science Collaboration 2015; Ebersole et al. 2016; Camerer et al. 2018; Silberzahn et al. 2018) and it has been argued that the inappropriate use of statistics is a leading cause of irreproducible…”
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings (2018, background): “The present report follows up on initial evidence from the Many Labs projects (Ebersole et al., 2016; Klein et al., 2014a).”
Choosing prediction over explanation in psychology: Lessons from machine learning (2017, background): “One alternative to conducting small-sample research would be for researchers to participate in large, multilab, collaborative projects (Ebersole et al., 2015; Klein et al., 2014).”
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences (2017, result): “For example, in the recent BMany labs 3^ project (Ebersole et al., 2016), which examined whether effects could be reproduced when the same procedure was run in multiple labs, the Stroop effect was repli cated in 100% of attempts, compared to much lower rates for most effects tested.”
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis (2018, background): “As a by e w o r U n number of authors have argued in recent years (Edwards & Berry 2010, Fiedler 2017, d e d can be used to develop more nuanced theoretical propositions and more effective interventions.”
Reproducible research in sport and exercise psychology: The role of sample sizes (2016, neutral): “…a crisis of confidence The so-called crisis of confidence, sometimes also called replication crisis centers around the observation that it has been surprisingly hard to replicate well-known psychological results (e.g., Ebersole et al., 2015; Nosek & Lakens, 2014; Open Science Collaboration, 2015).”
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education (2018, result): “Interestingly, when researchers have conducted replication studies, results have often failed to validate original findings (Camerer et al., 2016; Ebersole et al., 2016; Klein et al., 2014; Open Science Collaboration, 2015).”
Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence (2019, background): “Regarding research on social media, meta-analyses can scrutinize the stability of associations across, for example, different social media platforms, populations, or activities on SNSs (Ebersole et al., 2016; Klein et al., 2014; Klein et al., 2018; O’Donnell et al., 2018).”