State of the Art: Reproducibility in Artificial Intelligence

By Odd Erik Gundersen, Sigbjørn Kjensmo. AAAI Conference on Artificial Intelligence, 2018

353 citations recorded by the source corpus.

Background: Research results in artificial intelligence (AI) are criticized for not being reproducible. Objective: To quantify the state of reproducibility of empirical AI research using six reproducibility metrics measuring three different degrees of reproducibility. Hypotheses: 1) AI research is not documented well enough to reproduce the reported results. 2) Documentation practices have improved over time. Method: The literature is reviewed and a set of variables that should be documented to enable reproducibility are grouped into three factors: Experiment, Data and Method. The metrics describe how well the factors have been documented for a paper. A total of 400 research papers from the conference series IJCAI and AAAI have been surveyed using the metrics. Findings: None of the papers document all of the variables. The metrics show that between 20% and 30% of the variables for each factor are documented. One of the metrics show statistically significant increase over time while the others show no change. Interpretation: The reproducibility scores decrease with in- creased documentation requirements. Improvement over time is found. Conclusion: Both hypotheses are supported.

Subfields: Artificial Intelligence, Information Systems and Management, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/151c7cb45c75ac5573a6c03b2ea089b34512898c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Statistics and Probability

Citations in the corpus

Transparency and reproducibility in artificial intelligence (2020, background): “Even with extensive description, reproducing complex computational pipelines based purely on text is a subjective and challenging task 9 .”
Applications of deep learning in stock market prediction: recent progress (2020, methodology): “We also care more about the implementation workflow and result reproducibility of previous studies, e.g., dataset and code availability, which is a problem that has drawn the attention from the AI researchers (Gundersen & Kjensmo, 2018).”
Show Your Work: Improved Reporting of Experimental Results (2019, methodology): “In this work, we follow Gundersen and Kjensmo (2018) and use an extended notion of this concept: when comparing two methods, two research groups with different implementations should follow an experimental procedure which leads to the same conclusion about which performs better.”
Harnessing multimodal data integration to advance precision oncology (2021, background): “Reproducibility and benchmarking are major challenges in AI, with many published biomedical AI studies failing to provide the source code, test data or both 128 .”
The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research (2020, background): “Scholars argue that AI firms are less diverse than academia (West et al., 2019) and produce research that is less reproducible than academic research (Gundersen & Kjensmo, 2018).”
The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations (2021, methodology): “After conducting a survey of 400 algorithms presented in research papers at two top AI conferences (IJCAI and NeurIPS), researchers reported that only 6% of the presented papers shared the algorithm's code, a third shared the data on which they tested their algorithms, and only half shared a partial”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Since code availability is fundamental for replicability/reproducibility [34] 2 , we provide for each work its corresponding code repository, if available, so that interested readers can reproduce and learn from existing studies; • We conclude the paper by discussing in Section 6 a compelling…”