Variable Noise and Dimensionality Reduction for Sparse Gaussian processes

By Edward Snelson, Zoubin Ghahramani. 2006

84 citations recorded by the source corpus.

The sparse pseudo-input Gaussian process (SPGP) is a new approximation method for speeding up GP regression in the case of a large number of data points N. The approximation is controlled by the gradient optimization of a small set of M `pseudo-inputs', thereby reducing complexity from N^3 to NM^2. One limitation of the SPGP is that this optimization space becomes impractically big for high dimensional data sets. This paper addresses this limitation by performing automatic dimensionality reduction. A projection of the input space to a low dimensional space is learned in a supervised manner, alongside the pseudo-inputs, which now live in this reduced space. The paper also investigates the suitability of the SPGP for modeling data with input-dependent noise. A further extension of the model is made to make it even more powerful in this regard - we learn an uncertainty parameter for each pseudo-input. The combination of sparsity, reduced dimension, and input-dependent noise makes it possible to apply GPs to much larger and more complex data sets than was previously practical. We demonstrate the benefits of these methods on several synthetic and real world problems.

Subfields: Applied Mathematics, Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

1. Michael A. Osborne
2. Jan Peters
3. Oliver Stegle
4. Ryan P. Adams
5. Marc Peter Deisenroth
6. Philipp Hennig
7. Carl E. Rasmussen
8. Roberto Calandra

In Artificial Intelligence

1. Sergey Levine
2. Marc Peter Deisenroth
3. Jan Peters
4. Pieter Abbeel
5. Carl E. Rasmussen
6. Michael A. Osborne
7. Chelsea Finn
8. Dieter Fox

In Control and Systems Engineering

1. Marc Peter Deisenroth
2. Jan Peters
3. Sergey Levine
4. Carl E. Rasmussen
5. Roberto Calandra
6. Pieter Abbeel
7. Wolfram Burgard
8. Michael A. Osborne

Citations in the corpus

• Manifold Gaussian Processes for regression (2014, methodology): “We experimentally validate our mGP model on a discontinuous function, a function with multiple length scales, and on locomotion data from a real bipedal robot, where the effect of ground contacts makes standard GP regression challenging.”
• Most likely heteroscedastic Gaussian process regression (2007, background): “Re­cently, Snelson and Ghahramani (2006) proposed to utilize the dependency of the predictive uncertainty on the density of input data points.”
• Active Learning of Linear Embeddings for Gaussian Processes (2013, background): “The problem of discovering linear embeddings of GP s was discussed by Snelson & Ghahramani (2006) for the passive case.”
• Gaussian process product models for nonparametric nonstationarity (2008, methodology): “Wechose the SPGP for comparison to the GPPM, as it is one of the few 6 Gaussian Process Product Models 1.5 1 2 EP Posterior g(x) EP Posterior g(x) EP Posterior g(x) 1 0.5 1.8 0.5 0 1.6 0 -0.5 1.4 -0.5 -1 1.2 -1 -1.5 1 150 4 100 EP Posterior f(x)*exp(g(x)) EP Posterior f(x)*exp(g(x)) EP Posterior ”
• Gaussian Process Training with Input Noise (2011, background): “Snelson and Ghahramani [6] suggest a different approach whereby the importance of points in a pseudo-training set can be varied, allowing the posterior variance to vary as well.”