Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters

By Ziyun Wang, Nando de Freitas. 2014

100 citations recorded by the source corpus.

Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to find the best optima. Surprisingly, there is little theoretical analysis of this crucial problem in the literature. To address this, we derive a cumulative regret bound for Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expected improvement acquisition function and sub-Gaussian observation noise, provides us with guidelines on how to design hyper-parameter estimation methods. A simple simulation demonstrates the importance of following these guidelines.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sunil Gupta
2. Svetha Venkatesh
3. Santu Rana
4. Thanh Dai Nguyen
5. Stephan Günnemann

In Computational Theory and Mathematics

1. Sunil Gupta
2. Svetha Venkatesh
3. Santu Rana

In Management Science and Operations Research

1. Sunil Gupta
2. Svetha Venkatesh
3. Santu Rana

Citations in the corpus

• Stable Bayesian optimization (2018, background): “Convergence analyses of Bayesian optimization for EI [3], EI with noisymeasurement [22], and GP-UCB [20] provide guarantee of convergence to the optimum of the objective function.”