Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

By Evan C. Carter, Felix D. Schönbrodt, W. Gervais, Joseph Hilgard. Advances in Methods and Practices in Psychological Science, 2019

461 citations recorded by the source corpus.

Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/.

Subfields: Applied Psychology, Experimental and Cognitive Psychology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3ca5752ed11fe5d331af462326ea1e54af29e29b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Experimental and Cognitive Psychology

In Statistics and Probability

Citations in the corpus

Identification of and Correction for Publication Bias (2017, unknown): “Carter et al. (2017) compare di↵erent meta-analytic methods for psychological research.”
Meta-Analysis of Present-Bias Estimation using Convex Time Budgets (2019, methodology): “It has been argued that the performance of commonly used bias-correction methods such as the FAT-PET procedure depends on the nature of the data, and no single method dominates the other in all circumstances (Alinaghi and Reed, 2018; Carter et al., 2019; Hong and Reed, 2019).”
Sample Size Justification (2021, unknown): “Check if the bias detection tests that are reported in the meta-analysis are state-of-the-art, or perform multiple bias detection tests yourself (Carter, Schönbrodt, Gervais, & Hilgard, 2019), and consider bias corrected effect size estimates (even though these estimates might still be biased, and…”
Simulation-Based Learning in Higher Education: A Meta-Analysis (2020, methodology): “We expect high effects of simulation-based learning (i.e., Cook, 2014) and do not expect high publication bias; however, we expect high heterogeneity of the effects, which according to Carter et al. (2017) can hinder some types of analysis (e.g., p-curve analysis).”
Evaluation of a decided sample size in machine learning applications (2023, background): “Small studies have a lower probability of producing true effects because of the higher chance of type I or II error [2, 3].”