Analysis of the AutoML Challenge Series 2015-2018

By Isabelle M Guyon, Lisheng Sun-Hosoya, M. Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag and 3 more. 2019

189 citations recorded by the source corpus.

The ChaLearn AutoML Challenge (The authors are in alphabetical order of last name, except the first author who did most of the writing and the second author who produced most of the numerical analyses and plots.) (NIPS 2015 – ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed bya one-round AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This chapter analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this chapter (data and code) have been made publicly available at http://automl.chalearn.org/.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Survey on Automated Machine Learning (2019, methodology): “To solve some of these problems, the ChaLearn AutoML challenge (Guyon et al., 2015, 2016, 2018) has been introduced.”
Automated Machine Learning: From Principles to Practices (2018, methodology): “The technical roadmap of NAS is shown in Figure 19, which reveals the trade-offs between search performance and search speed for different NAS methods.”
OpenML Benchmarking Suites (2017, methodology): “The AutoML benchmark explicitly sources part of their datasets from the OpenML-CC18, but also includes datasets used in AutoML competitions (primarily Guyon et al. [2019]) or previous comparisons of AutoML systems.”
Practical Automated Machine Learning for the AutoML Challenge 2018 (2018, background): “The second AutoML challenge (Guyon et al., 2018.) just finished, and with this paper we describe our winning entry to this challenge, PoSH Auto-sklearn.”
Automated Machine Learning in Practice: State of the Art and Recent Results (2019, methodology): “This inspired the series of Automated Machine Learning (AutoML) challenges since 2015 [33].”
Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, methodology): “1, 4 and S1 depict the (normalized/raw) accuracy and inference-latency of our distilled models (under the GIB-1 strategy which is superior to others), compared with both the teacher (AutoGluon ensemble), as well as ensembles produced by H2O-AutoML [7] and AutoSklearn [2], two popular AutoML framewor”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “We evaluate TabTransformer and baseline models on 15 publicly available binary classification datasets from the UCI repository (Dua and Graff 2017), the AutoML Challenge (Guyon et al. 2019), and Kaggle (Kaggle, Inc.”
AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, methodology): “Numerous AutoML frameworks have recently emerged to solve this problem, as evidenced by a flurry of survey articles (Yao et al., 2018; He et al., 2019; Truong et al., 2019; Guyon et al., 2019; Gijsbers et al., 2019; Zöller & Huber, 2019).”