An Open Source AutoML Benchmark

By P. Gijsbers, E. LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, Joaquin Vanschoren. 2019

257 citations recorded by the source corpus.

In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “More specifically, a combination of the curated benchmarking suites OpenML1006 (Bischl et al., 2017), OpenML-CC187 (Bischl et al., 2019) and AutoML Benchmark8 (Gijsbers et al., 2019) is used.”
Survey on Automated Machine Learning (2019, methodology): “standardized format, have been selected. More specically, a combination of the curated benchmarking suites OpenML1005 (Bischl et al., 2017), OpenML-CC186 (Bischl et al., 2019) and AutoML Benchmark7 (Gijsbers et al., 2019) is used. The combination of these benchmarking suits contains 137 classication”
OpenML Benchmarking Suites (2017, methodology): “The AutoML benchmark [Gijsbers et al., 2019] also makes use of an OpenML benchmark suite to evaluate AutoML tools in a reproducible manner.”
Techniques for Automated Machine Learning (2019, background): “In contrast, several benchmarks [23; 3; 30; 101] have been published for AutoMHT for performance comparison.”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “In Section 5, we additionally use the AutoML benchmark (Gijsbers et al., 2019) to evaluate Auto-sklearn 2.”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “For instance, OpenML benchmarks CC-18, CC-100, [Bischl et al., 2021] and AutoML Benchmark Gijsbers et al. [2019] contain tabular data, but also include images and artiﬁcial datasets, which may explain why they have not been used in tabular deep learning papers.”
Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, methodology): “Gaussian We initially considered additional classiﬁcation datasets from Gijsbers et al. [39], but decided to not to include those for which: it was trivial to get near 100% accuracy for many model types (so a teacher is unnecessary), the data are dominated by missing values, the original data are…”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “In this section, we study the differences between the set of datasets we selected for our benchmarks with the set selected by previous benchmarks, namely the CC18 benchmark [Bischl et al., 2021] and the AutoML benchmark [Gijsbers et al., 2019].”
A Hierarchy of Limitations in Machine Learning (2020, background): “This is a drawback to benchmark data sets; while it is incredibly important to have common data and measures of success to make sure that competing methods are compared on the same grounds, after a certain point any improvements to the state of the art may be overfitting (Gijsbers et al., 2019) (ano”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Figure 2 shows attention maps on the MNIST [82] and volkert [29] data sets, which can be found using different attention mechanisms.”
FLAML: A Fast and Lightweight AutoML Library (2019, methodology): “Our main empirical study is based on a combination of a recent open source AutoML classification benchmark (Gijsbers et al., 2019) and a regression benchmark (Olson et al.”
Transformers Can Do Bayesian Inference (2021, methodology): “We used a large collection of tabular datasets from the open-source OpenML AutoML Benchmark (Gijsbers et al., 2019); we ﬁrst removed datasets with more than one hundred features or missing values, ending up with 20 datasets that represent a diverse set of classiﬁcation problems with numerical and…”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “…experiments (538 650 total models trained), we limited to datasets smaller than 1.1M. CC-18 and OpenML Benchmarking Suite are both seen as the go-to standards for conducting a fair, diverse evaluation across algorithms due to their rigorous selection criteria and wide diversity of datasets [6, 25].”
AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, methodology): “Actual training times of each framework in the AutoML Benchmark, which varied despite the fact that we instructed each framework to only run for the listed time limit.”
Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, methodology): “This was accomplished in 2019 with the contribution of an open source benchmark to compare AutoML algorithms on a dataset of 39 classification tasks [49].”