Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

By Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter. 2016

620 citations recorded by the source corpus.

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed Fabolas, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that Fabolas often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Others have proposed to estimate performance on small data (Klein et al., 2017) or model sizes (Zoph et al., 2018; Real et al., 2019) in the context of neural architecture search (NAS).”
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets (2017, methodology): “This could be exploited by multi-ﬁdelity methods for architecture and hyperpa-rameter search (Li et al., 2016; Klein et al., 2016).”
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, background): “…been a recent line of work in hyperparam-eter optimization (HPO) (Snoek et al., 2012; Hutter et al., 2011; Bergstra et al., 2011; Li et al., 2017; Klein et al., 2017a; Falkner et al., 2018) 1 and neural architecture search (NAS) (Baker et al., 2017; Zoph and Le, 2017; Real et al., 2017; Elsken…”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Fabolas (Klein et al., 2016) treats the budget s as an additional input parameter in the search space that can be freely chosen by the optimization procedure instead of being deterministically calculated.”
Survey on Automated Machine Learning (2019, background): “Algorithm 2 hyperband Input: bmin, bmax, Λ 1: bmax ← log ⌊ bmax bmin ⌋ 2: for b ∈ {bmax, bmax − 1, . . . , 0} do 3: m← DetermineBudget(b) 4: ~λ← SampleConfigurations(m,Λ) 5: SuccessiveHalving(~λ) 6: end for An alternative to SuccessiveHalving and hyperband is Fabolas (Klein et al., 2016).”
Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “In traditional hyperparameter optimization, it is standard for empirical results to be reported over 10 independent experimental runs [14, 25, 23, 27].”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Finally, Klein et al. (2016) recently introduced Fabolas, a new Bayesian optimization method that combines adaptive selection and evaluation.”
Population Based Training of Neural Networks (2017, methodology): “For this reason, a number of methods look to speed up the updating of the posterior with information from each optimisation process by performing early stopping, using intermediate losses to predict final performance, and even modelling the entire time and data dependent optimisation process in orde”
A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “Tuning an SVM using the same search space as (Klein et al., 2017a).”
Massively Parallel Hyperparameter Tuning (2018, methodology): “, 2017), two state-of-the-art methods that exploit partial training (Klein et al., 2017; Jaderberg et al., 2017).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “These acquisition functions can be used for selective subsample evaluation (Klein et al., 2017), reducing the number of necessary resampling iterations (Swersky et al., 2013), and stopping certain model classes, such as NNs, early.”
Automatic Discovery of Privacy–Utility Pareto Fronts (2019, methodology): “For non-private single-objective optimization, there is an abundance of recent work in machine learning on hyperparameter selection, using BO [23, 26] or other methods [32] to maximize a model’s utility.”
Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization (2016, methodology): “While we intended to compare H YPERBAND with Fabolas, we encountered some technical difﬁculties when using the package 3 and are working with the authors of Klein et al. (2016) to resolve the issues.”
Bayesian optimization for conditional hyperparameter spaces (2017, methodology): “[10] model the function of validation error with regards to dataset size and let the Bayesian optimization procedure choose the subset size with relevant acquisition functions.”