Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters

By Katharina Eggensperger. 2013

366 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Neural Architecture Search & AutoML

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Since GP-BO methods do not work well on such conﬁguration spaces (Eggensperger et al., 2013) we do not include them in this experiment.”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “(2014) on the benchmark set of continuous problems from the HPOLib package (Eggensperger et al., 2013).”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Many probability models could be used in the BO algorithm (Eggensperger et al., 2013), but Gaussian process (GP) (Rasmussen, 2003) is overwhelmingly the most widely used.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “HPOlib (Eggensperger et al., 2013) aims to provide standardized data sets for the evaluation of CASH algorithms.”
Survey on Automated Machine Learning (2019, neutral): “HPOlib (Eggensperger et al., 2013) aims to provide standardized data sets for the evaluation of CASH algorithms.”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “We do not include Gaussian process based Bayesian optimization meth-ods (Shahriari et al., 2016), such as, e.g., Spearmint (Snoek et al., 2012), since this tends to have problems in high-dimensional discrete optimization tasks (Eggensperger et al., 2013).”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “However, these methods tackle a fundamentally challenging problem of simultaneously ﬁtting and optimizing a high-dimensional, non-convex function with unknown smoothness, and possibly noisy evaluations.”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, methodology): “We used HPOLIB to run the experiments for all optimizers with a single format, both for the original hyperparameter optimization benchmarks and for our surrogates.”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, background): “Due to the conditional hyperparameters in the scikit-learn space we excluded Spearmint, which – without modification – is known to perform poorly in their presence (Eggensperger et al. 2013).”
Input Warping for Bayesian Optimization of Non-Stationary Functions (2014, methodology): “Our experiments on the continuous subset of the HPOLib benchmark (Eggensperger et al., 2013) shows that input warping performs substantially better than state-ofthe-art baselines on these problems.”
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn (2014, methodology): “On Convex Shapes, hyperopt-sklearn outperforms previous automatic algorithm configuration approaches [Egg13] and manual tuning [Lar07] .”
CMA-ES for Hyperparameter Optimization of Deep Neural Networks (2016, neutral): “, 2011) are known to perform better (Bergstra et al.; Eggensperger et al., 2013; Domhan et al., 2015), here we focus on continuous optimization.”
Tune: A Research Platform for Distributed Model Selection and Training (2018, neutral): “HyperOpt, Spearmint, and HPOLib (Snoek et al. (2012); Eggensperger et al. (2013)) are distributed model selection tools that manage both the search and evaluation of the model, implementing search techniques such as random search and tree of parzen estimators (TPE).”
Best Practices for Scientific Research on Neural Architecture Search (2019, background): “Although this might be an ambitious goal in view of the young field, other metadomains showed that it is in fact feasible and worthwhile (Eggensperger et al., 2013; Hutter et al., 2014; Bischl et al., 2016).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “Eggensperger et al. (2013) showed that (i) BO with GPs were a strong approach for small continuous spaces with few evaluations, (ii) BO with Random Forests performed well for mixed spaces with a few hundred evaluations, and (iii) for large spaces and many evaluations, ES were the best optimizers.”
An Efficient Approach for Assessing Hyperparameter Importance (2014, background): “…we are most interested in random forest models, since these have been shown to achieve the best performance for model-based optimization in complex hyperparameter spaces, particularly in cases involving categorical and conditional hyperparameters (Thornton et al., 2013; Eggensperger et al., 2013).”
Automated Machine Learning: Methods, Systems, Challenges (2019, positive): “Because of the increased usage of machine learning in companies, HPO is also of substantial commercial interest and plays an ever larger role there, be it in companyinternal tools [45], as part of machine learning cloud services [6, 89], or as a service by itself [137].”
Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization (2016, background): “Existing empirical evidence suggests that these methods outperform random search (Thornton et al., 2013; Eggensperger et al., 2013; Snoek et al., 2015).”