Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

By Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Thomas Lindauer, Frank Hutter. Journal of machine learning research, 2020

404 citations recorded by the source corpus.

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

View Paper: https://www.semanticscholar.org/paper/3bddc8f2dff60d9da7d7c5f315f3467966f61ff1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “To aggregate results across datasets of varying difﬁculty, we use a metric similar to the distance to the minimum (or average distance to the minimum (ADTM) when averaged across datasets), used in Feurer et al. [2021] and introduced in Wistuba et al. [2015].”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “We also include results for F1-score and ROC AUC in Appendix D. Similar to prior work [20, 62], whenever we average across datasets, we use the average distance to the minimum (ADTM) metric, which consists of 0-1 scaling (after selecting the best hyperparameters, which helps protect against…”
OpenML Benchmarking Suites (2017, methodology): “OpenML offers bindings with the Java, Python and R ecosystems [van Rijn, 2016, Feurer et al., 2021b, Casalicchio et al., 2017] to provide easy integration in common machine learning tools, workﬂows, and environments.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “Furthermore, our work shares the general goal with automated machine learning (AutoML) [90, 91] in improving the usability of machine learning, though LIFT uses only a single pretrained LMs for all tasks while AutoML automates the standard machine learning pipeline from a set of existing algorithms.”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, methodology): “To aggregate results across datasets of varying difficulty, we use a metric similar to the distance to the minimum (or average distance to the minimum –ADTM– when averaged across datasets), used in Feurer et al. [2021] and introduced in Wistuba et al. [2015].”