Benchmark and Survey of Automated Machine Learning Frameworks
By Marc-André Zöller, Marco F. Huber. 2019
449 citations recorded by the source corpus.
Machine learning (ML) has become a vital part in many aspects of our daily life. However, building well performing machine learning applications requires highly specialized data scientists and domain experts. Automated machine learning (AutoML) aims to reduce the demand for data scientists by enabling domain experts to automatically build machine learning applications without extensive knowledge of statistics and machine learning. This paper is a combination of a survey on current AutoML methods and a benchmark of popular AutoML frameworks on real data sets. Driven by the selected frameworks for evaluation, we summarize and review important AutoML techniques and methods concerning every step in building an ML pipeline. The selected AutoML frameworks are evaluated on 137 different data sets.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank Hutter
- Marco F. Huber
- Marc-André Zöller
- Bernd Bischl
- Matthias Feurer
- Jan N. van Rijn
- Joaquin Vanschoren
- Mu Li
In Evaluation & Benchmarks
- Frank Hutter
- Matthias Feurer
- Bernd Bischl
- Jan N. van Rijn
- Giuseppe Casalicchio
- Joaquin Vanschoren
- Li Yang
- Abdallah Shami
In Neural Architecture Search & AutoML
- Matthias Feurer
- Jan N. van Rijn
- Bernd Bischl
- Zhenguo Li
- Joaquin Vanschoren
- Li Yang
- Abdallah Shami
- Yujun Li
Citations in the corpus
- On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “Secondly, they are only efficient for convex functions because the local instead of a global optimum may be reached for non-convex functions [2].”
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, methodology): “Prior work focused almost exclusively on the task of Combined Algorithm Selection and Hyperparameter optimization (CASH), offering strategies to find the best model and its hyperparameters from a sea of possibilities (Thornton et al., 2013; Zöller & Huber, 2019).”
- Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning (2020, methodology): “To alleviate this problem, we take advantage of automated machine learning (AutoML)(He et al., 2019; Z¨oller and Huber, 2019; Elshawi et al., 2019) to automatically adapt hyper parameters.”
- OpenML Benchmarking Suites (2017, methodology): “Sometimes more datasets are used, as Zöller and Huber [2021] combine it with OpenML-CC18 and OpenML100 and Kadra et al. [2021] add datasets from UCI and Kaggle.”
- Grid search in hyperparameter optimization of machine learning models for prediction of HIV/AIDS test results (2021, background): “Therefore, the following process needs to be done manually to identify the global optimums [23]:”
- An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, background): “…learning, where not only traditional hyperparameters such as learning rates (Zeiler, 2012), but also neural architectures (Liu et al., 2019), data augmentation policies (Cubuk et al., 2018) and other plausible variables are to be set automatically (Elsken et al., 2019; Zöller and Huber, 2019).”