The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or Is It Something Else?

By Frank L. Schmidt, In-Sue Oh. 2016

123 citations recorded by the source corpus.

There have been frequent expressions of concern over the supposed failure of researchers to conduct replication studies. But the large number of meta-analyses in our literatures shows that replication studies are in fact being conducted in most areas of research. Many who argue for replication as the “gold standard” consider a non-significant replication attempt to be strong evidence against the initial study, an interpretation that ignores statistical power, typically low in behavioral research. Many researchers also hold that there is no need to replicate a non-significant finding, believing it will always replicate perfectly, an erroneous belief. These beliefs lead to a widely accepted sequential model of the research process that is deficient because it assumes that a single study can answer a research question, a belief that meta-analysis has shown to be false. Meta-analysis can provide the solution to these problems if the problems of publication bias and questionable research practices are successfully addressed. The real problem is not a lack of replication; it is the distortion of our research literatures caused by publication bias and questionable research practices.

Subfields: General Health Professions, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/244f7e211fa620f3f474cbd831214484386c8112

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Statistics and Probability

Citations in the corpus

Near and Far Transfer in Cognitive Training: A Second-Order Meta-Analysis (2019, result): “Naïve (i.e., uncorrected) meta-analytic means are often less reliable than the publication-bias corrected estimates (Schmidt & Oh, 2016; Stanley, 2017).”
Advancing the social identity theory of leadership: A meta-analytic review of leader group prototypicality (2020, background): “…of the impact of leader group prototypicality on leader outcomes by providing an assessment that encompasses the entire body of existing research evidence (Cumming, 2014; Schmidt & Oh, 2016) while also exploring the potential influence of publication bias (Banks et al., 2012; Kepes et al., 2012).”
Answers to 18 Questions About Open Science Practices (2018, unknown): “Second, the sharing of design protocols, measures, and analytic scripts can help to improve the rigor of study designs (Nosek et al., 2015) as well as reproducibility and replication rates of success (Open Science Collaboration, 2015; Schmidt & Oh, 2016).”
Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence (2019, background): “Scholars in this interdisciplinary field need to take seriously prior calls for reporting studies irrespective of the results’ effect size or statistical significance, being transparent about research data, and ending questionable research practices (Schmidt & Oh, 2016).”
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field. (2018, background): “To some extent, we agree with Schmidt and Oh (2016)’s argument that rather than increasing replication, other issues such as publication bias and QRPs need to be tackled first and then meta-analyses could address the lack of direct replication (see Coyne, 2016 for related arguments; and Schimmack,…”
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis (2018, background): “Rather, it is a common result as evidence accumulates (Schmidt & Oh 2016).”