Initializing Bayesian Hyperparameter Optimization via Meta-Learning

By Matthias Feurer, Jost Tobias Springenberg, Frank Hutter. 2015

492 citations recorded by the source corpus.

Model selection and hyperparameter optimization is crucial in applying machine learning to a novel dataset. Recently, a subcommunity of machine learning has focused on solving this problem with Sequential Model-based Bayesian Optimization (SMBO), demonstrating substantial successes in many applications. However, for computationally expensive algorithms the overhead of hyperparameter optimization can still be prohibitive. In this paper we mimic a strategy human domain experts use: speed up optimization by starting from promising configurations that performed well on similar datasets. The resulting initialization technique integrates naturally into the generic SMBO framework and can be trivially applied to any SMBO method. To validate our approach, we perform extensive experiments with two established SMBO frameworks (Spearmint and SMAC) with complementary strengths; optimizing two machine learning frameworks on 57 datasets. Our initialization procedure yields mild improvements for low-dimensional hyperparameter optimization and substantially improves the state of the art for the more complex combined algorithm selection and hyperparameter optimization problem.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, methodology): “Various kinds of methods have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradientbased optimization (Bengio, 2000; Domke, 2012; Macla”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “In this paper, we focus only on combining Hyperband and Bayesian optimization, but we would like to mention that methods improving BO are potentially applicable to BOHB as well, such as meta learning (Swersky et al., 2013; Feurer et al., 2015; Poloczek et al., 2016; Springenberg et al., 2016), activ”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, background): “Recent work has sought to improve the performance of the GP approach through accommodating higher dimensional problems (Wang et al., 2013; Djolonga et al., 2013), input non-stationarities (Snoek et al., 2014) and initialization through meta-learning (Feurer et al., 2015).”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Auto-Sklearn auto-sklearn (Feurer et al., 2015a, 2018) is a tool for building classification and regression pipelines.”
Survey on Automated Machine Learning (2019, methodology): “auto-sklearn (Feurer et al., 2015a, 2018) is a tool for building classification and regression pipelines.”
Transfer Learning with Neural AutoML (2018, methodology): “These include random search [1], parameter modeling [3], metalearned hyperparameter initialization [16], deep-learning based tree searches over a predefined model-specification language [17], and learning of gradient descent optimizers [18, 19].”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “property within the Bayesian optimization framework [12], [49], [79], [89], [148], [160].”
Learning to Optimize (2016, methodology): “Most methods [13, 4, 24, 25, 9] rely on sequential model-based Bayesian optimization [22, 6], while others adopt a random search approach [3] or use gradient-based optimization [2, 8, 21].”
SmartML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning for Machine Learning Algorithms (2019, background): “In our framework, the meta-learning feature is emulating the role of the domain expert in the field of machine learning [4, 11].”
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization (2017, methodology): “A number of publications show that using meta-data is beneficial, including but not limited to [2, 16,52,61,66,69,71].”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “In this work, we describe in detail how we improved Auto-sklearn from the first version (Feurer et al., 2015a) to construct PoSH Auto-sklearn, which won the second competition and then describe how we improved PoSH Auto-sklearn further to yield our current approach for Auto-sklearn 2.0 .”
Meta-QSAR: a large-scale application of meta-learning to drug design and discovery (2017, methodology): “One approach is to ﬁnd a good set of initial candidate conﬁgurations by using meta-learning (Feurer et al., 2015): based on meta-features, one can ﬁnd the most similar datasets and use the optimal algorithms and parameter settings for these datasets as the initial candidates to evaluate.”
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates (2017, methodology): “…spaces with categorical parameters and high degrees of conditionality also exist in HPO, e.g., in the optimization of machine learning frameworks (Thornton et al 2013; Feurer et al 2015a) or architectural optimization of deep neural networks (Bergstra et al 2014; Domhan et al 2015).”
P4ML: A Phased Performance-Based Pipeline Planner for Automated Machine Learning (2018, methodology): “While Feurer et al. (2015a) use random forests as a surrogate, recent advances have been achieved by using matrix factorization (Fusi et al., 2017) or scaling Gaussian process surrogates to large collections of metadata (Wistuba et al., 2018).”
Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start (2017, background): “…work, we are interested in such a setting as we would like to warm start BO by, e.g., transferring information obtained from previous runs of the BO routine, or learn across similar problems (e.g., a given classiﬁer applied across different datasets [2, 25, 8, 9, 10]), which we will call tasks .”