Joint Optimization and Variable Selection of High-dimensional Gaussian Processes

By Rui M. Castro, Andreas Krause. 2012

92 citations recorded by the source corpus.

Maximizing high-dimensional, nonconvex functions through noisy observations is a notoriously hard problem, but one that arises in many applications. In this paper, we tackle this challenge by modeling the unknown function as a sample from a high-dimensional Gaussian process (GP) distribution. Assuming that the unknown function only depends on few relevant variables, we show that it is possible to perform joint variable selection and GP optimization. We provide strong performance guarantees for our algorithm, bounding the sample complexity of variable selection, and as well as providing cumulative regret bounds. We further provide empirical evidence on the effectiveness of our algorithm on several benchmark optimization problems.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, positive): “[39] made significant progress by introducing a two stage strategy for optimization and variable selection of high-dimensional GPs.”
Active Learning of Linear Embeddings for Gaussian Processes (2013, methodology): “Within Bayesian optimization, much recent work has focused on high-dimensional problems (Hutter et al., 2011; Chen et al., 2012; Carpentier & Munos, 2012; Bergstra & Bengio, 2012; Hutter, 2009).”
Bayesian Optimization in a Billion Dimensions via Random Embeddings (2013, background): “Finally, one important advantage of REMBO is that — in contrast to the approach of Chen et al. (2012) — it does not require the eﬀective dimension to be coordinate aligned.”
An Efficient Approach for Assessing Hyperparameter Importance (2014, background): “Recent work on Bayesian optimization has targeted the case where most hyperparameters are truly unimportant (Chen et al., 2012; Wang et al., 2013), and several applications have yielded evidence that some hyperparameters indeed tend to be much more important than others (Bergstra and Bengio, 2012;…”
Optimization as Estimation with Gaussian Processes in Bandit Settings (2015, methodology): “When dealing with high-dimensional problems, one common kind of assumption is that the function to be optimized has an underlying low-dimensional structure, which can be solved using traditional Bayesian optimization methods [8, 9, 43, 18].”
UvA-DARE (Digital Academic Repository) Bayesian Optimization in High Dimensions via Random Embeddings (2013, unknown): “For reference, we also evaluated the method of [Chen et al., 2012] for these functions, confirming that it does not handle rotation gracefully: while it performed best in the non-rotated case for D = 25, it performed worst in the rotated case.”
Scalable Global Optimization via Local Bayesian Optimization (2019, background): “There is a considerable body of work in high-dimensional BO [8, 21, 5, 44, 14, 45, 32, 26, 27, 6].”
Stable Bayesian optimization (2018, background): “While this simple strategy is powerful for many applications, there have been recent attempts to make it widely applicable by making it feasible in high dimension [4], and adding the ability to perform transfer learning [10], multi-objective optimization [11], batch optimization [1,16], etc.”
Recent Advances in Bayesian Optimization (2022, methodology): “In [23] two strategies, the inite diference sequential likelihood ratio test and the GP sequential likelihood ratio test, are proposed to screen the most contributing variables.”
Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters (2013, background): “As a result, hyperparameter optimization has become an active research area within Bayesian optimization, with characteristics such as low effective dimensionality [1, 5, 6] and problem variants, such as optimization across different data sets [7] being explored.”
High Dimensional Bayesian Optimisation and Bandits via Additive Models (2015, background): “To our knowledge, most literature for GPB/ BO in high dimensions are in the setting where the function varies only along a very low dimensional subspace (Chen et al., 2012; Wang et al., 2013; Djolonga et al., 2013).”
Bayesian Multi-Scale Optimistic Optimization (2014, background): “Examples of acquisition functions include probability of improvement, expected improvement, Bayesian expected losses, upper confidence bounds (UCB), and dynamic portfolios of these (Močkus, 1982; Jones, 2001; Garnett et al., 2010; Srinivas et al., 2010; Chen et al., 2012; Hoffman et al., 2011).”
High-Dimensional Gaussian Process Bandits (2013, background): “In [18] the more general case of functions sampled from a GP under the same sparsity assumptions was considered.”
Bayesian Optimization and Semiparametric Models with Applications to Assistive Technology (2014, background): “Chen et al. [2012] explore the case where the input is considered sparse and one wishes to select the relevant input dimensions while performing the optimization.”