Replicability is not Reproducibility:Nor is it Good Science

By C. Drummond. 2009

296 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Anserini (2018, background): “Just focusing on the computational sciences (in particular, setting aside the medical sciences), there has already been much discussion [21, 24, 37, 44] as well as concrete proposals to improve the state of affairs [20, 25, 46].”
Building a Science of Individual Differences from fMRI. (2016, methodology): “Reproducibility consists in obtaining the same results using the same code and data [184] (Figure I) (see also [183] for a slightly different terminology); it is a lower scientific standard than fully independent replication using a different dataset and similar analysis, but ensuring reproducibilit”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “The absence of community norms like reproducibility guidance [34, 114], documentation standards [98] or statistical signiﬁcance testing [16] makes relying on outcomes-based approaches to evaluate progress even more questionable [13].”
Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies (2015, background): “Drummond [35] responds to Blockeel and Vanschoren by highlighting the practical difficulty of an experiment database: ”[s]urely we wouldn’t expect reviewers to very carefully study the scripts etc needed to produce the results.”
Research-paper recommender systems: a literature survey (2015, unknown): “Replicability describes an exact copy of an experiment that uses the same tools, follows the same steps, and produces the same results [296].”
Offspring from Reproduction Problems: What Replication Failure Teaches Us (2013, background): “Most researchers agree that validating results entails that a method should lead to the same over-all conclusions rather than producing the exact same numbers (Drummond, 2009; Dalle, 2012; Buchert and Nussbaum, 2012, etc.).”
Reproducibility in forecasting research (2015, methodology): “For example, Drummond (2009) used the terms in the opposite way to the above definitions.”
archivist: An R Package for Managing, Recording and Restoring Data Analysis Results (2017, background): “…gets increasing attention in the academic literature across various disciplines, see for example Peng (2009) for bioinformatics or Koenker and Zeileis (2009) for the econometric research or Drummond (2009) for more general discussion about diﬀerences between replicability and reproducibility.”
Trustworthy AI: From Principles to Practices (2021, methodology): “To obtain a clear assessment, degrees of reproducibility have been studied in such works as the ACM Artifact Review and References [106, 143].”
Best Practices in Data Analysis and Sharing in Neuroimaging using MRI (2016, unknown): “While some have argued that reproducibility is secondary, and that “one should replicate the result not the experiment” [Drummond2009], recent failures to replicate high-impact results and occasional but acutely concerning examples of outright fraud have made the case for the importance of…”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “Reproducibility in machine learning As numerous works have pointed out [Goodman et al., 2016, Gunder-sen and Kjensmo, 2018, Barba, 2018, Drummond, 2009], the word reproducibility can correspond to very different standards, ranging from the ability to reproduce statistically similar values [Raff,…”
On the Reuse of Scientific Data (2017, methodology): “Slightly different is a replication study, where novel data are used to ask an old question using the same data and analysis methods (Drummond 2009).”