Input Warping for Bayesian Optimization of Non-Stationary Functions

By Jasper Snoek, Kevin Swersky, Richard S. Zemel, Ryan P. Adams. 2014

265 citations recorded by the source corpus.

Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate the effects of spatially-varying length scale. We develop a methodology for automatically learning a wide family of bijective transformations or warpings of the input space using the Beta cumulative distribution function. We further extend the warping framework to multi-task Bayesian optimization so that multiple tasks can be warped into a jointly stationary space. On a set of challenging benchmark optimization tasks, we observe that the inclusion of warping greatly improves on the state-of-the-art, producing better results faster and more reliably.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Generative Moment Matching Networks (2015, methodology): “…and GMMN training, dropout rate for the auto-encoder, and number of hidden units on each layer of both auto-encoder and GMMN, were tuned using Bayesian optimization (Snoek et al., 2012; 2014) 1 to optimize the validation set likelihood under the Gaussian Parzen window density estimation.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “In our experiments with soft attention, we also used Whet-lab 1 (Snoek et al., 2012; 2014) in our Flickr8k experiments.”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “A comparison of the time per suggested experiment for our method compared to the state-of-the-art GP based approach (Snoek et al., 2014) on the six dimensional Hartmann function.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “rates, weight decay, etc. [1]. The poor scaling of naïve methods like grid search with dimensionality has driven interest in more sophisticated hyperparameter optimization methods over the past years [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. Bayesian optimization has emerged as an efﬁcient framework, achiev”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, methodology): “One strategy is to convert a stationary kernel into a nonstationary one by transforming x using a parametric warping function xðwÞ 1⁄4 wðxÞ and then applying a stationary kernel to xðwÞ [129], [145].”
Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “Figure 5 shows the loss compared with a random baseline, Spearmint with length scale fixed to the training length scale, and Spearmint with automatic inference of the GP hyper-parameter values and input warping to deal with non-stationarity [Snoek et al., 2014].”
Manifold Gaussian Processes for regression (2014, methodology): “Speciﬁcally, we use the squared exponential covariance function with Automatic Rele-vance Determination (ARD) with , where l i are the characteristic length-scales, and σ 2 f is the variance of the latent function F .”
Freeze-Thaw Bayesian Optimization (2014, methodology): “In this section, we empirically validate our method by comparing to the state-of-the-art (Warped GP EI MCMC) Bayesian optimization method of [20] on three common machine learning tasks: Online LDA, Logistic Regression and Probabilistic Matrix Factorization.”
Non-stochastic Best Arm Identification and Hyperparameter Optimization (2015, methodology): “Although several hyperparameter optimization methods have been proposed recently, e.g., Snoek et al. (2012; 2014); Hutter et al. (2011); Bergstra et al. (2011); Bergstra & Bengio (2012), the vast majority of them consider the training of machine learning models to be black-box procedures, and only…”
Multi-task Neural Networks for QSAR Predictions (2014, methodology): “In order to have an experimental protocol that avoids as many of the viccissitudes of human expert judgement as possible, we set all neural net metaparameters using Bayesian optimization [Snoek et al., 2012, 2013] of the validation AUC.”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “, using random forests [27], Gaussian processes [65] and input warped Gaussian processes [67] or scalable Bayesian optimization [66].”
Recent Advances in Bayesian Optimization (2022, methodology): “In addition, input space warpings or non-linear mappings are used to remove the non-stationary efects in a latent space [170].”
Learning to Learn without Gradient Descent by Gradient Descent (2016, methodology): “We compare our learning to learn approach with popular state-of-the-art Bayesian optimization packages, including Spearmint with automatic inference of the GP hyperparameters and input warping to deal with non-stationarity (Snoek et al., 2014), Hyperopt (TPE) (Bergstra et al., 2011), and SMAC (Hutte”
Towards Further Automation in AutoML (2018, methodology): “Recent advancements in hyperparameter optimization (Snoek et al., 2012, 2014, 2015; Hutter et al., 2011; Shahriari et al., 2016) triggered the development of new AutoML systems based on them, e.g., Auto-WEKA (Thornton et al., 2013), hyperopt-sklearn (Komer et al., 2014) and Auto-sklearn (Feurer et…”
Constrained Bayesian Optimization with Noisy Experiments (2017, methodology): “Spearmint EI uses similar heuristics as EI+heuristics to handle noise, but also uses a diﬀerent approach for GP estimation, diﬀerent optimization routines, and other techniques like input warping (Snoek et al. 2014).”
A Stratified Analysis of Bayesian Optimization Methods (2016, methodology): “We used the HIPS implementation of Spearmint (Snoek et al., 2014a).”
Learning to Optimize: A Primer and A Benchmark (2021, background): “[53] demonstrated that their learned RNN optimizers are competitive with state-of-the-art Bayesian optimization packages (Spearmint [184], SMAC [94], and TPE [22]).”
Heteroscedastic Treed Bayesian Optimisation (2014, methodology): “s not place explicit requirements on its GP leaves so long as we can compute the log-marginal-likelihood and EI e- ciently. As a result, not only could we use standard GPs but also GPs with warping (Snoek et al., 2013) as leaves. When using warping, the and parameters of the Beta CDF are estimated u”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “[89] Jasper Snoek, Kevin Swersky, Richard Zemel, and Ryan P Adams.”
Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets (2016, background): “To some degree this can be addressed by warping the input space [24], although a more elaborate treatment suggests the adoption of computationally demanding deep GP hierarchies [35].”