On doing relevant and rigorous experiments: Review and recommendations
By Sirio Lonati, Bernardo F. Quiroga, Christian Zehnder, John Antonakis. Journal of Operations Management, 2018
317 citations recorded by the source corpus.
Abstract Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state-of-the-science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top-tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best-practice methodological suggestions (i.e., the “ten commandments” of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi-experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.
Subfields: Economics and Econometrics, General Decision Sciences, Management Information Systems
View Paper: https://www.semanticscholar.org/paper/4c6e39e1103304b1fca5c8c76a5eb5e5018a7a8e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Economics and Econometrics
In General Decision Sciences
In Management Information Systems
- Lutz Kaufmann
- Stephanie Eckerd
- Craig R. Carter
- Elliot Bendoly
- Mohammad Asif Salam
- Robert D. Klassen
- Amrou Awaysheh
- Helen Walker
Citations in the corpus
- On making experimental design choices: Discussions on the use and challenges of demand effects, incentives, deception, samples, and vignettes (2020, methodology): “These “out-of-task” perceptual measurements often are the underlying drivers of observed behaviors (Ajzen, 1991; Gino & Pisano, 2008), and can be evaluated using vignette study designs (or non-consequential decision making, in Lonati et al., 2018, and addressed in their fourth “commandment”).”