Gaussian Processes for Regression and Optimisation

By Phillip Boyle. 2007

72 citations recorded by the source corpus.

Gaussian processes have proved to be useful and powerful constructs for the purposes of regression. The classical method proceeds by parameterising a covariance function, and then infers the parameters given the training data. In this thesis, the classical approach is augmented by interpreting Gaussian processes as the outputs of linear filters excited by white noise. This enables a straightforward definition of dependent Gaussian processes as the outputs of a multiple output linear filter excited by multiple noise sources. We show how dependent Gaussian processes defined in this way can also be used for the purposes of system identification. Onewell known problem with Gaussian process regression is that the computational complexity scales poorly with the amount of training data. We review one approximate solution that alleviates this problem, namely reduced rank Gaussian processes. We then show how the reduced rank approximation can be applied to allow for the efficient computation of dependent Gaussian processes. We then examine the application of Gaussian processes to the solution of other machine learning problems. To do so, we review methods for the parameterisation of full covariance matrices. Furthermore, we discuss how improvements can be made by marginalising over alternative models, and introduce methods to perform these computations efficiently. In particular, we introduce sequential annealed importance sampling as a method for calculating model evidence in an on-line fashion as new data arrives. Gaussian process regression can also be applied to optimisation. An algorithm is described that uses model comparison between multiple models to find the optimum of a function while taking as few samples as possible. This algorithm shows impressive performance on the standard control problem of double pole balancing. Finally, we describe how Gaussian processes can be used to efficiently estimate gradients of noisy functions, and numerically estimate integrals.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “Since EGO’s publication, there has evolved a body of work devoted to extending the algorithm, particularly in adding constraints to the optimization problem [6], [131], [23], and in modelling noisy functions [14], [75], [76].”
Kernels for Vector-Valued Functions: a Review (2011, methodology): “Boyle and Frean [14, 15] introduced the process convolution approach for multiple outputs to the machine learning community with the name of “dependent Gaussian processes” (DGP), further developed in [13].”
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010, positive): “…has evolved a body of work devoted to extending the algorithm, particularly in adding constraints to the optimization problem [Audet et al., 2000, Sasena, 2002, Boyle, 2007], and in modelling noisy functions [Bartz-Beielstein et al., 2005, Huang et al., 2006, Hutter et al., 2009, Hutter, 2009].”
An experimental methodology for response surface optimization methods (2011, background): “…surface methods has been steady over the past 40 years (Kushner 1964; Perttunen et al. 1989; Stuckman 1989; Mockus 1989; Sacks et al. 1989; Elder 1992; Cox and John 1995; Schonlau 1997; Jones and Schonlau 1998; Gutmann 2001; Sasena 2002; Santner et al. 2003; Boyle 2006; Regis 2007; Lizotte 2008).”
Portfolio Allocation for Bayesian Optimization (2010, unknown): “As such, Bayesian optimization techniques are some of the most efficient approaches in terms of the number of function evaluations required [30, 23, 25, 3, 5].”
Modeling an augmented Lagrangian for improved blackbox constrained optimization (2014, methodology): “For a wider review of the literature, we recommend the tutorial by Brochu et al. (2010) and the Ph.D. thesis of Boyle (2007), both on “Bayesian optimization.””