Efficient Benchmarking of Hyperparameter Optimizers via Surrogates

By Katharina Eggensperger, Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown. 2015

145 citations recorded by the source corpus.

Hyperparameter optimization is crucial for achieving peak performance with many machine learning algorithms; however, the evaluation of new optimization techniques on real-world hyperparameter optimization problems can be very expensive. Therefore, experiments are often performed using cheap synthetic test functions with characteristics rather different from those of real benchmarks of interest. In this work, we introduce another option: cheap-to-evaluate surrogates of real hyperparameter optimization benchmarks that share the same hyperparameter spaces and feature similar response surfaces. Specifically, we train regression models on data describing a machine learning algorithm’s performance depending on its hyperparameter setting, and then cheaply evaluate hyperparameter optimization methods using the model’s performance predictions in lieu of running the real algorithm. We evaluated a wide range of regression techniques, both in terms of how well they predict the performance of new hyperparameter settings and in terms of the quality of surrogate benchmarks obtained. We found that tree-based models capture the performance of several machine learning algorithms well and yield surrogate benchmarks that closely resemble real-world benchmarks, while being much easier to use and orders of magnitude cheaper to evaluate.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Evaluation & Benchmarks

In Neural Architecture Search & AutoML

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “Surrogate-based method [121]–[123] is another powerful tool that approximates to the black box function.”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “For the next experiments we constructed a set of surrogate benchmarks based on offline data following Eggensperger et al. (2015).”
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, positive): “Recent work (Eggensperger et al., 2015) proposed to use surrogate benchmarks, which replace the original benchmark by a regression model trained on data generated offline.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “, (Snoek et al., 2012; Eggensperger et al., 2015; Klein et al., 2017a).”
Survey on Automated Machine Learning (2019, background): “, (Snoek et al., 2012; Eggensperger et al., 2015; Klein et al., 2017)— are not suited as a benchmark as no significant differences can be observed.”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “The resulting data deﬁnes a tabular surrogate benchmark (Eggensperger et al., 2015) that enables NAS experiments to be run via querying a table for arbitrary architectures instead of performing the usual costly train and evaluate procedure.”
Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “Especially in such cases, we advocate running extensive evaluations on tabular benchmarks, such as NAS-Bench-101 (Ying et al., 2019) and the NAS-HPO benchmarks (Klein & Hutter, 2019), or on surrogate benchmarks as proposed by Eggensperger et al. (2015; 2018) and used by Falkner et al. (2018).”
Automated Machine Learning: From Principles to Practices (2018, background): “For reducing the number of data samples, several works consider to sample a subset of the dataset [62, 71, 100, 112].”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “Our principal methodology is inspired by Eggensperger et al. (2015), who balanced unbiased collection by uniform random sampling for good overall coverage with biased and dense data collection in high-performance regions by running hyperparameter optimizers.”
A Stratified Analysis of Bayesian Optimization Methods (2016, methodology): “…facilitate the transfer of insights garnered from experimentation on these functions to real-world problems, following the path of (Bergstra et al., 2014; Eggensperger et al., 2015) which used cheap test functions and surrogates of hyperparameter problems to evaluate Bayesian optimization software.”
Hyperparameter Importance Across Datasets (2017, unknown): “depth [1, 10] iterations [50, 500] imputation {mean, median, mode}”
Fast Bayesian hyperparameter optimization on large datasets (2017, methodology): “surrogate (Eggensperger et al., 2015) on our SVM grid on MNIST (described in Section 3), for which we had performed all function evaluations beforehand.”
Efficient Parameter Importance Analysis via Ablation with Surrogates (2017, methodology): “…algorithm configuration, they were already a key technique in perinstance algorithm selection (see, e.g., Rice 1976, Kotthoff 2014); furthermore, they have recently been used to define surrogate benchmarks for efficient benchmarking of hyperparameter optimization tools (Eggensperger et al. 2015).”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, methodology): “• Surrogate evaluator: For configurations that can be readily quantized, one straightforward method to cut down the evaluation cost is to build a model that predicts the performance of given configurations, with experience of past evaluations [38], [54], [96], [97].”
Practical Hyperparameter Optimization for Deep Learning (2018, methodology): “For the feedforward network, we constructed a surrogate benchmark following Eggensperger et al. (2015); optimizing a surrogate instead of the real objective function is substantially cheaper, allowing us to afford many independent runs and draw statistically meaningful conclusions.”
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates (2017, methodology): “(3) Since we already know that random forests are well suited as EPMs (Hutter et al 2014b; Eggensperger et al 2015), we use a quantile regression forest (QRF; Meinshausen 2006) for the quantile regression.”