Enhancing reproducibility for computational methods

By Victoria Stodden, M. McNutt, David H. Bailey, Ewa Deelman, Yolanda Gil, B. Hanson, M. Heroux, John P. A. Ioannidis, Michela Taufer. 2016

396 citations recorded by the source corpus.

Data, code, and workflows should be available and cited Over the past two decades, computational methods have radically changed the ability of researchers from all areas of scholarship to process and analyze data and to simulate complex systems. But with these advances come challenges that are contributing to broader concerns over irreproducibility in the scholarly literature, among them the lack of transparency in disclosure of computational methods. Current reporting methods are often uneven, incomplete, and still evolving. We present a novel set of Reproducibility Enhancement Principles (REP) targeting disclosure challenges involving computation. These recommendations, which build upon more general proposals from the Transparency and Openness Promotion (TOP) guidelines (1) and recommendations for field data (2), emerged from workshop discussions among funding agencies, publishers and journal editors, industry participants, and researchers representing a broad range of domains. Although some of these actions may be aspirational, we believe it is important to recognize and move toward ameliorating irreproducibility in computational research.

Subfields: Artificial Intelligence, Information Systems, Information Systems and Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Systems and Management

Citations in the corpus

Opportunities and obstacles for deep learning in biology and medicine (2017, methodology): “We strongly advocate following established best practices for sharing source code, archiving code in repositories that generate digital object identifiers, and open licensing [410] regardless of the minimal requirements, or lack thereof, set by journals, conferences, or preprint servers.”
On doing better science: From thrill of discovery to policy implications (2017, background): “To the extent possible, authors should consider posting data on permanent repositories (Stodden et al., 2016).”
Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination (2021, methodology): “…setting of identifiable functionals and real-valued outcomes, the arguments put forth by Dimitriadis et al. (2021) continue to apply, in that CORP estimators are bound to, simultaneously, improve statistical efficiency, reproducibility (Stodden et al., 2016), and stability (Yu and Kumbier, 2020).”
Reproducing GW150914: The First Observation of Gravitational Waves From a Binary Black Hole Merger (2020, background): “This notion of reproducibility is magnified both in importance and challenges in the context of computational science workflows.(1) An increasingly large fraction of scientific results depend on computational elements, which in turn creates reproducibility challenges associated with the implementati”
NMRbox: A Resource for Biomolecular NMR Computation. (2017, background): “Although the problem of reproducibility is enjoying recent attention (13), and the problem of software persistence has been raised (14), the resource described here appears to be the only active effort to address the preservation of the functional computational environment needed to reproduce a comp”
Freesound Datasets: A Platform for the Creation of Open Audio Datasets (2017, methodology): “After the observations reported in section 1 and by embracing the ideas described in [22] and [33] for sustainable MIR evaluation and reproducibility of computational methods, we deﬁne the following principles that apply to our vision of Freesound Datasets and the creation of datasets with the…”
Better together: Elements of successful scientific software development in a distributed collaborative community (2020, background): “The general lack of documentation or example workflows (“protocol captures”[8,9]) further diminishes reproducibility and usability[10,11].”
Systematic integration of biomedical knowledge prioritizes drugs for repurposing (2017, background): “Our goals were to contribute to the advancement of science ( Hrynaszkie-wicz, 2011 ; Molloy, 2011 ), maximize our impact ( McKiernan et al., 2016 ; Piwowar and Vision, 2013 ), and enable reproducibility ( Stodden et al., 2016 ; Stodden and Miguez, 2014 ; Bagg-erly, 2010 ).”