The Meaning of Failed Replications: A Review and Proposal
By Michael A. Clemens. Social Science Research Network, 2015
207 citations recorded by the source corpus.
Economists are increasingly using publicly shared data and code to check each other’s work, an exercise often called ‘replication’ testing. But this much-needed trend has not been accompanied by a consensus about what ‘replication’ means. If a follow-up study does not ‘replicate’ an original result, according to current usage of the term, this can mean anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes an unambiguous definition of replication. Many social scientists already use the term in the way suggested here, but many more do not. The paper contrasts this definition with decades of unsuccessful attempts to standardize terminology, and argues that many prominent results described as replication tests should not be described as such. It argues that professional associations should formally adopt this definition, thereby improving incentives for researchers to conduct more and better replication tests.
Subfields: Computer Networks and Communications, Management of Technology and Innovation
View Paper: https://www.semanticscholar.org/paper/c61458f97dc0352a22e427487f4da301ddf5dc9b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Management of Technology and Innovation
Citations in the corpus
- Volatility estimation for Bitcoin: Replication and robustness (2019, background): “1 However, there is no consensus on definitions for the concepts of reproducibility, replicability and robustness (see, e.g., Arulampalam et al., 1997; Pesaran, 2003; Hamermesh, 2007; Clemens, 2017), which can lead to confusion.”
- Identification of and Correction for Publication Bias (2017, unknown): “By a “replication” we mean what Clemens (2017) terms a “reproduction,” obtained by applying the same experimental protocol or analysis to a new sample from the same population as the original study. For each published X in a given set of studies, such replications provide an independent estimateX go”
- The influence of hidden researcher decisions in applied microeconomics (2020, unknown): “…allow us to determine variation in results that arises from the use of different samples, when testing the same hypothesis in new data, or from major analysis choices, when testing the same hypothesis in the same data with a new method (Christensen & Miguel, 2018; Clemens, 2017; Hamermesh, 2007).”