The case for open computer programs

By Darrel C. Ince, L. Hatton, John Graham-Cumming. 2012

548 citations recorded by the source corpus.

Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

Citations in the corpus

Albumentations: fast and flexible image augmentations (2018, background): “Building open source software enhances the rigor and impact of research by allowing others to reproduce published results [19,54].”
ITK: enabling reproducible research and open science (2014, unknown): “…the true practice of the scientiﬁc method and publish the full details of their methodology, including the source code, data, parameters and auxiliary documents that are required for a third party to independently repeat the work and verify its ﬁndings (Ince et al., 2012; Collins and Tabak, 2014).”
Describing human decisions in agent-based models - ODD + D, an extension of the ODD protocol (2013, background): “Finally, the category “Implementation Details” was included in the Details block because we believe that this information will improve comparability and reproducibility (see also Ince et al., 2012).”
SimpleElastix: A User-Friendly, Multi-lingual Library for Medical Image Registration (2016, background): “Scientific source code is typically not published [16], is lacking documentation or is difficult to use because it has not been written with other researchers in mind.”
Game On, Science - How Video Game Technology May Help Biologists Tackle Visualization Challenges (2013, background): “In the context of open computer programs [34], this approach provides users and developers not only with access to the source code but also with a very accessible interface (i.e. the editor) to examine the underlying science and implementation of the program.”
The secret lives of experiments: Methods reporting in the fMRI literature (2012, unknown): “Finally, because no standardized reporting recommendations can ever capture all relevant methodological details for all studies, investigators should be encouraged to release the source code underlying experimental tasks and analysis techniques (Ince et al., 2012).”
Advances in global change research require open science by individual researchers (2012, background): “Revealing one’s potentially imperfect and inelegant computer code and related details may be uncomfortable at first, but the community must accept that computational decisions should be subject to review and refinement much like the rest of our science (Ince et al., 2012).”
OpenQuake Engine: An Open Hazard (and Risk) Software for the Global Earthquake Model (2014, methodology): “The development of open-source software in this framework (Lees, 2012) is therefore a prerequisite, which allows seismic-hazard and risk practitioners to scrutinize and contribute to the methodologies/algorithms adopted for the calculations.”
A Novel, Open Access Method to Assess Sleep Duration Using a Wrist-Worn Accelerometer (2015, methodology): “The (visual) screening of accelerometer (actigraphy) data and sleep log data can be cumbersome in large datasets and it complicates replication [22].”
The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) (2015, background): “18 It is the latter open source aspect of our work which signiﬁcantly motivated our participation in BRATS 2013 as it provides a reproducible and publicly available framework for performing such an important task in neuroimaging [121].”
Data-analysis strategies for image-based cell profiling (2017, background): “Researchers can capture and share the computational environment used to create the results, such as providing virtual machines or Docker containers.”