Unreproducible Research is Reproducible

By Xavier Bouthillier, César Laurent, Pascal Vincent. 2019

129 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “This lack of rigour leads to contributions with apparent new state-of-the-art results, which are evaluated under different dataset splits, evaluation metrics, experimental budgets [43], and other uncontrolled sources of variations [43, 3].”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “data augmentations or training regularization) pose a challenge to the reproducibility of such methods, which is of great concern in the machine learning community in general (Bouthillier et al., 2019).”
The Benchmark Lottery (2021, background): “[Machado et al., 2018, Bouthillier et al., 2021, Balduzzi et al., 2018, Bouthillier et al., 2019, Musgrave et al., 2020], or “What are the best practices for fair benchmarking?””
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[11] Xavier Bouthillier, César Laurent, and Pascal Vincent.”
Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program) (2020, methodology): “, 2017), or even when using different random parameter initializations ((Bouthillier et al., 2019; Henderson et al., 2018).”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “…values [Raff, 2019, McDer-mott et al., 2019, Thavasimani and Missier, 2016], to successfully executing code [Collberg and Proebsting, 2016], to the ability to reproduce a relative relationship (a model remains state of art even when the experimental set-up is changed) [Bouthillier et al., 2019].”