Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

By Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, Hugo Larochelle. 2020

607 citations recorded by the source corpus.

One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.

Subfields: Artificial Intelligence, Information Systems and Management, Materials Chemistry, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Materials Chemistry

In Evaluation & Benchmarks

Citations in the corpus

BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “…and collected conversations with interested researchers in order to make this research accessible and reproducible, and thus to enable further research into responsible conversational AI (Sonnenburg et al., 2007; Pineau et al., 2021; Zhang et al., 2022; Roller et al., 2020; Dinan et al., 2021).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “This has led to many state-of-the-art agents incorporating numerous heterogeneous components, contributing to their increased complexity and to growing concerns about the reproducibility of research (Pineau et al., 2020).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “18653/v1/n18-1202 [105] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle.”
The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, methodology): “For example, NeurIPS’19 introduced the ML Reproducibility checklist for submitted papers (Pineau et al., 2020) which was also adopted by EMNLP’20 and”
CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability (2021, background): “These works mainly encompass a check-list for quantitative experiments (Pineau et al., 2021), the construction of elaborate dataset sheets (Gebru et al.”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “Furthermore, academic ML conferences such as NeurIPS have instituted reproducibility checklists for paper submissions (Pineau et al., 2020).”
Experience Replay with Likelihood-free Importance Weights (2020, methodology): “Our implementations are all based on RLkit † which gives a fair comparison between the baselines and our approach given the difﬁculty to reproduce the best RL results [29, 10].”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “In recent years, a significant effort has been made in robotics to address challenges in benchmarking [267] and reproducibility [268].”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[79] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle.”
Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training (2021, methodology): “Although the DL community is aware of the nondeterminism caused by using different random seeds [72, 90] , such algorithmic factors are not the sole source of DL nondeterminism.”
Between Subjectivity and Imposition (2020, methodology): “Fortunately, the machine learning research community has begun to promote similar reflexive practices: Following Pineau’s suggestion [69], authors of NeurIPS and ICML conference are now requested to include a reproducibility checklist which encourages “a complete description of the data collection p”
Using Unity to Help Solve Intelligence (2020, background): “This has become a pressing subject in several scientiﬁc ﬁelds (Baker, 2016; Dodge et al., 2019), including within the reinforcement learning community (Henderson et al., 2017; Pineau et al., 2020).”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “There is a reliance on self-reported results with no trust guarantees [75], and many state-of-the-art claims are famously difﬁcult to reproduce [52].”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “…evidence-based medicine where systematic reviews follow a strict elaboration protocol [71], or, more recently, in the machine learning community where authors are encouraged to follow pre-established checklists or datasheets to communicate their models and datasets effectively [2, 6, 29, 49, 55].”
The Computational Limits of Deep Learning (2020, background): “the reproducibility checklists of ICML [59] and NeurIPS[93, 106]), few papers in the other benchmark areas provide sufficient information to analyze the computational burden via network operations.”
The Fallacy of AI Functionality (2022, background): “In fact, many of the current critiques, policy positions and interventions in algorithmic accountability implicitly begin from the premise that such deployed algorithmic systems work, echoing narratives of super-human ability [62], broad applicability [149], and consistency [145], espoused in corpor”
Trustworthy AI: From Principles to Practices (2021, methodology): “Reproducibility checklists have been recently widely adopted in ML conferences to assess the reproducibility of submissions [263].”