Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

By Kevin Swersky, David Kristjanson Duvenaud, Jasper Snoek, Frank Hutter, Michael A. Osborne. 2014

84 citations recorded by the source corpus.

In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Neural Architecture Search & AutoML

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “[120] for tackling the neural architecture search problem.”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “While there has been some work on BO for architecture search [2, 15, 25, 37, 41], they have only been applied to optimise feed forward structures, e.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “Recent work has focused on defining a fixed-length embedding of conditional spaces where a standard kernel using Euclidean distance can be applied [147].”
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells (2018, methodology): “Scaling up Bayesian methods for architecture search is an ongoing work, and few kernel-based approaches have already shown solid performance [14, 41].”
Automated Machine Learning: From Principles to Practices (2018, methodology): “[175] designs a new search space to enable better transfer ability among different datasets.”
Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters (2013, background): “These conditional parameters are, for example, common in deep architectures, or in frameworks including many alternative algorithms, and some hyperparameter optimizers exploit knowledge about these conditionalities in their models to improve their performance [9, 8, 11, 12].”
Bayesian Optimization with Tree-structured Dependencies (2017, methodology): “Several authors attempted to design covariance functions that are aware of the structure: Duvenaud et al. (2011) consider kernels with an additive structure, while Swersky et al. (2014a); Hutter & Osborne (2013) introduce the Arc-kernel.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “Extensions for mixed-hierarchical spaces that are based on special kernels (Swersky et al., 2014) exist, and the use of random embed-dings has been suggested for high-dimensional spaces (Nayebi et al., 2019; Wang et al., 2016).”
Beyond Manual Tuning of Hyperparameters (2015, background): “Some, but not all, hyperparameter optimizers exploit knowledge about conditional hyperparameters to improve their performance [5, 32, 64].”
Learnable Embedding Space for Efficient Neural Architecture Compression (2019, methodology): “BO has been used to tune the number of layers and the size of hidden layers (Bergstra et al., 2011; Swersky et al., 2014), the width of a network (Snoek et al., 2012) or the size of the ﬁlter bank (Bergstra et al., 2013), along with other hyper-parameters, such as the learning rate, number of…”
Towards Automatically-Tuned Neural Networks (2016, methodology): “Adapting GPs to handle these characteristics is an active field of research (Swersky et al., 2013; Wang et al., 2016), but so far Bayesian optimization methods using tree-based models (Hutter et al.”
Extrapolating Learning Curves of Deep Neural Networks (2014, methodology): “Automating this manual trick would be valuable for speeding up both structure search (Bergstra et al., 2011, 2013; Swersky et al., 2013) and hyperparameter optimization (Snoek et al., 2012; Eggensperger et al., 2013), which are currently often prohibitively expensive (Krizhevsky et al., 2012).”
Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves (2015, result): “SVM [Coates et al., 2011] 4000 20 .40% SVM [Coates et al., 2011] 1600 22.10% DNN [Swersky et al., 2013] Table 2: Comparison of classification results on the k-means features extracted from the CIFAR10 dataset for different optimizers in comparison to previously published results. seeds and show…”
Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA (2017, background): “Conditional dependencies can also be accommodated in the Gaussian process framework (Hutter and Osborne, 2013; Swersky et al., 2013), but currently, tree-based methods achieve better performance. budget available for the entire learning process.”