What does research reproducibility mean?

By Steven N. Goodman, Daniele Fanelli, John P. A. Ioannidis. Science Translational Medicine, 2016

1,120 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3728c289e3582f6329a735b3e9882b2a0cabad83

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[34] Steven N Goodman, Daniele Fanelli, and John PA Ioannidis.”
Unreproducible Research is Reproducible (2019, methodology): “In this work we will use the terms proposed by Goodman et al. (2016), which avoid the ambiguity of the terms reproducibility, replicability and repeatability.”
Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness (2020, result): “…is associated with results reproducibility (that is, the production of corroborating results in a new study, having followed the same experimental methods 65 ), whether it is a consequence of the inadequate reporting observed in the modelling literature, 59 or other related issues is unclear.”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “Reproducibility in machine learning As numerous works have pointed out [Goodman et al., 2016, Gunder-sen and Kjensmo, 2018, Barba, 2018, Drummond, 2009], the word reproducibility can correspond to very different standards, ranging from the ability to reproduce statistically similar values [Raff,…”
The reproducibility crisis in the age of digital medicine (2019, methodology): “” For our purposes here, we take reproducibility to mean “obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible” (also known as “research reproducibility” or simply “replicability”).(3) A sense of crisis its”
State of the Art: Reproducibility in Artificial Intelligence (2018, methodology): “Replication, as used by (Drummond 2009) and (Stodden 2011), is in line with methods reproducibility as proposed by (Goodman et al. 2016) while reproducibility seems to entail both results reproducibility and inferential reproducibility.”
A standard protocol for reporting species distribution models (2020, methodology): “In the the lexicon of research reproducibility (Goodman et al. 2016), methods reproducibility means that sufficient details are provided on data and methods in order to independently repeat the study, while results reproducibility means that the same results can be obtained from an independent…”
Reproducibility vs. Replicability: A Brief History of a Confused Terminology (2018, background): “To solve the terminology confusion, Goodman et al. (2016) propose a new lexicon for research reproducibility with the following definitions: • Methods reproducibility: provide sufficient detail about procedures and data so that the same procedures could be exactly repeated.”
Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research* (2019, result): “…the claims made are in fact true by replicating the study (i.e., conducting the same study but with a new sample/dataset) or by attempting to reproduce identical findings using the same data and procedures as the initial study (Ketokivi and Mantere, 2010; Goodman, Fanelli, and Ioannidis, 2016).”
CONSORT 2025 statement: updated guideline for reporting randomised trials (2025, positive): “50 We encourage trial authors to detail what was done and to acknowledge if something was not done or was modified, ensuring alignment of information with that reported in the trial protocol, statistical analysis plan, and trial registry.”
The ARRIVE guidelines 2.0: updated guidelines for reporting animal research (2020, background): “In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers (Begley & Ioannidis, 2015, Goodman et al. 2016).”
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research* (2020, neutral): “Why good reporting is important In recent years, concerns about the reproducibility of research findings have been raised by scientists, funders, research users, and policy makers [1, 2].”