MCMC for Variationally Sparse Gaussian Processes

By James Hensman, Alexander G. de G. Matthews, Maurizio Filippone, Zoubin Ghahramani. 2015

149 citations recorded by the source corpus.

Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper is available at github.com/sparseMCMC.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “The robustmax is deﬁned as [24, 19]: Note that as we have distributions over the latent variables the gradient is able to ﬂow through these max selecting operations and the likelihood can be evaluated through one-dimensional Gaussian quadrature.”
Stochastic Variational Deep Kernel Learning (2016, methodology): “Besides KLSP-GP, we also collected the results of the MC-GP [9] which uses a hybrid Monte Carlo sampler to tackle non-conjugate likelihoods, SAVI-GP [7] which approximates with a univariate Gaussian sampler,”
GPflow: A Gaussian Process Library using TensorFlow (2016, methodology): “As a scenario, we studied training a multiclass GP classiﬁer on MNIST using stochastic variational inference (Hensman et al., 2015b,a) 2 .”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, methodology): “We obtain The ﬁnal (univariate) expectation of the log-likelihood can be computed analytically in some cases, with quadrature (Hensman et al., 2015) or through Monte Carlo sampling (Bonilla et al., 2016; Gal et al., 2015).”
Multi-fidelity classification using Gaussian processes: accelerating the prediction of large-scale computational models (2019, methodology): “A future direction to reduce the cost of training the classiﬁer is to use variational inference [9, 10] instead of Markov chain Monte Carlo.”
A Scalable Hierarchical Gaussian Process Classifier (2019, methodology): “Recently, stochastic optimization has been successfully adopted for GP classification to cater for large-scale problems [24], [39].”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “hyperparameters [85]–[87] rather than traditional point estimation which risks overfitting when the number of hyperparameters is small and 2) the non-Gaussian likelihoods [86], [88], [89].”
Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation (2019, methodology): “Hensman et al. (2015) and Salimbeni et al. (2018) scale this model to big datasets employing a variational inference approach but rely on numerical quadrature.”