Evaluating replicability of laboratory experiments in economics

By Colin F. Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan and 8 more. Science, 2016

1,110 citations recorded by the source corpus.

Another social science looks at itself Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science, this issue p. 1433 By several metrics, economics experiments do replicate, although not as often as predicted. The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.

Subfields: Economics and Econometrics, General Decision Sciences, Safety Research

View Paper: https://www.semanticscholar.org/paper/81da5bd490672b1fbf64826a11888e6c59a6f61e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In General Decision Sciences

In Safety Research

Citations in the corpus

Best Practices in Data Collection and Preparation: Recommendations for Reviewers, Editors, and Authors (2019, background): “But, this is not unique to organizational research, and the same challenge is currently faced by economics (Camerer et al., 2016) and the natural sciences as well (Baker, 2016).”
Transparency, Reproducibility, and the Credibility of Economics Research (2018, positive): “One of the most important recent studies is Camerer et al. (2016), which repeated 18 behavioral economics lab experiments originally published between 2011 and 2014 in the American Economic Review and the Quarterly Journal of Economics to assess their replicability.”
What Motivates Effort?: Evidence and Expert Forecasts (2018, neutral): “The NLS model, as (12) stresses, matches the expected log effort, while the minimum-distance matches the log of expected effort (given the assumed homogeneity).”
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education (2018, result): “Interestingly, when researchers have conducted replication studies, results have often failed to validate original findings (Camerer et al., 2016; Ebersole et al., 2016; Klein et al., 2014; Open Science Collaboration, 2015).”
A Unified Welfare Analysis of Government Policies* (2019, positive): “To place these estimates in perspective, Andrews and Kasy (Forthcoming) use data from Camerer et al. (2016) to estimate that lab experiments in economics are 26-35 times more likely to be published if they have statistically significant results; in contrast, they estimate that studies of the minimum”
The Behavioral Revolution and International Relations (2017, background): “Camerer et al. 2016; Open Science Collaboration 2015.”
Variability in the analysis of a single neuroimaging dataset by many teams (2019, neutral): “18 (9), Brazil (3), Canada (16), China (2), Finland (1), France (4), Germany (21), Italy (13), the Netherlands (8), Spain (4), Sweden (4), Switzerland (3), Taiwan (3), UK (7) and the USA (76).”
Enhancing Rigor in Quantitative Entrepreneurship Research (2019, background): “fallibility of research findings (e.g., Baker, 2016; Camerer et al., 2016; Gelman, 2018; Honig et al., 2018; Miguel et al., 2014; Open Science Collaboration, 2015), insufficient replication and replicability (Bettis et al., 2016), and the growing maturity of the field.”
Proxies and Databases in Financial Misconduct Research (2017, positive): “similar to, or lower than, the replication rates recently reported in such other sciences as psychology, biomedicine, and experimental economics (see Open Science Collaboration 2015; Iqbal, Wallach, Khoury, Schully, and Ioannidis 2016; Camerer et al. 2016).”
The Replicability Crisis and Public Trust in Psychological Science (2018, neutral): “%) cases (Camerer et al., 2016).”