Benchmarking Automatic Machine Learning Frameworks

By A. Balaji, A. Allen. 2018

78 citations recorded by the source corpus.

AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source AutoML solutions using open source datasets. We test auto-sklearn, TPOT, auto_ml, and H2O's AutoML solution against a compiled set of regression and classification datasets sourced from OpenML and find that auto-sklearn performs the best across classification datasets and TPOT performs the best across regression datasets.

Subfields: Computer Networks and Communications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Balaji and Allen (2018) and Gijsbers et al. (2019) evaluate various AutoML frameworks on real data sets.”
Techniques for Automated Machine Learning (2019, background): “In contrast, several benchmarks [23; 3; 30; 101] have been published for AutoMHT for performance comparison.”
AVATAR - Machine Learning Pipeline Evaluation Using Surrogate Model (2020, background): “These tools are selected because their abilities to construct and optimise hyperparameters of complex ML pipelines have been empirically proven to be effective in a number of previous studies [2,5,16].”
An Open Source AutoML Benchmark (2019, methodology): “Prior eﬀorts to create systematic AutoML benchmarks were sadly obfuscated by memory management and evaluation setup issues and lack strong baselines to interpret the results (Balaji and Allen, 2018).”