OpenML Benchmarking Suites

By Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, R. G. Mantovani, Jan N. van Rijn, Joaquin Vanschoren. 2017

232 citations recorded by the source corpus.

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. Therefore, we advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites are (a) easy to use through standardized data formats, APIs, and client libraries; (b) machine-readable, with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We also present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “More specifically, a combination of the curated benchmarking suites OpenML1006 (Bischl et al., 2017), OpenML-CC187 (Bischl et al., 2019) and AutoML Benchmark8 (Gijsbers et al., 2019) is used.”
Survey on Automated Machine Learning (2019, methodology): “At first, all previously mentioned CASH algorithms are tested on the complete OpenML100 suite.”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “For instance, OpenML benchmarks CC-18, CC-100, [Bischl et al., 2021] and AutoML Benchmark [Gijsbers et al., 2019] contain tabular data, but also include images and artificial datasets, which may explain why they have not been used in tabular deep learning papers.”
Hyperparameters and tuning strategies for random forest (2018, methodology): “We use the OpenML100 benchmarking suite (Bischl et al., 2017) and download it via the OpenML R package (Casal-icchio et al., 2017).”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “…experiments (538 650 total models trained), we limited to datasets smaller than 1.1M. CC-18 and OpenML Benchmarking Suite are both seen as the go-to standards for conducting a fair, diverse evaluation across algorithms due to their rigorous selection criteria and wide diversity of datasets [6, 25].”
An Open Source AutoML Benchmark (2019, methodology): “Datasets We selected 39 classiﬁcation datasets from previous AutoML papers (Thornton et al., 2013), competitions (Isabelle et al., 2016), and machine learning benchmarks (Bischl et al., 2017a) 3 , according to a predeﬁned list of criteria.”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, methodology): “Datasets As test datasets, we used all datasets from the curated open-source OpenML-CC18 benchmark suite (Bischl et al., 2021) that contain up to 2 000 samples (1 000 for the training split), 100 features and 10 classes.”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, background): “(2019), which consists of datasets for comparing classifiers (Bischl et al., 2021) and datasets from the AutoML challenges (Guyon et al.”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “Other benchmarks use different metrics to remove too easy datasets, like removing datasets which can be learnt perfectly by a single decision classiﬁer [Bischl et al., 2021], but this does not account for different Bayes rate of different datasets.”