Fast Bayesian hyperparameter optimization on large datasets

By Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter. 2017

67 citations recorded by the source corpus.

Bayesian optimization has become a successful tool for optimizing the hyperparameters of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed FABOLAS, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that FABOLAS often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Figure 4 compares BOHB to various BO methods, such as Fabolas (Klein et al., 2017a), multi-task Bayesian optimization (MTBO) (Swersky et al., 2013), GP-BO with expected improvement (Snoek et al., 2012; Klein et al., 2017b), RS and HB. BOHB achieved similar performance as Fabolas and worked slightly…”
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, neutral): “…a deep neural network, more advanced Bayesian optimization methods try to speed up the optimization process by exploiting available ﬁdelities of the objective function, such as learning curves (Domhan et al.; Klein et al., 2017b; Falkner et al., 2018) or dataset subsets (Klein et al., 2017a).”
Machine learning and applications in ultrafast photonics (2020, background): “One can then continuously increase the volume of training data until the validation error stagnates.”
Practical Hyperparameter Optimization for Deep Learning (2018, methodology): “To obtain good anytime performance, a recent trend in BO is to extend the traditional blackbox setting by exploiting cheaper fidelities of the objective function (Swersky et al., 2014; Klein et al., 2017a; Swersky et al., 2013; Kandasamy et al., 2017).”
Analysis of the AutoML Challenge Series 2015-2018 (2019, methodology): “Some of these methods consider jointly the two-level optimization and take time cost as a critical guidance for hyper-parameter search (Swersky et al., 2014; Klein et al., 2017).”