Gaussian Processes for Big Data

By James Hensman, Nicoló Fusi, Neil D. Lawrence. 2013

1,384 citations recorded by the source corpus.

We introduce stochastic variational inference for Gaussian process models. This enables the application of Gaussian process (GP) models to data sets containing millions of data points. We show how GPs can be variationally decomposed to depend on a set of globally relevant inducing variables which factorize the model in the necessary manner to perform variational inference. Our approach is readily extended to models with non-Gaussian likelihoods and latent variable models based around Gaussian processes. We demonstrate the approach on a simple toy problem and two real world data sets.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “…2004; Barber and Chiappa 2006; Archambeau et al. 2007a, 2007b; Johnson and Willsky 2014; Foti et al. 2014), robust models (Tipping and Lawrence 2005; Wang and Blei 2015), and Gaussian process models (Titsias and Lawrence 2010; Damianou, Titsias, and Lawrence 2011; Hensman, Fusi, and Lawrence 2013).”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “One interesting line of work, for example, would be to introduce a similar methodology by instead employing the sparse Gaussian process as the underlying probabilistic model (Snelson & Ghahramani, 2005; Titsias, 2009; Hensman et al., 2013).”
Neural-net-induced Gaussian process regression for function approximation and PDE solution (2018, background): “In future work, we intend to leverage scalable GPs recently developed in [22, 23, 24] to tackle large dataset problems.”
Deep Gaussian Processes (2012, background): “Recall that each r(m)Xh and g (m) Y term is associated with a different GP and, thus, is coming with its own set of automatic relevance determination (ARD) weights (described in equation (5)).”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “Sparse Variational Gaussian processes Among various sparse approximations of Gaussian processes (Quiñonero- Candela & Rasmussen, 2005; Bui et al., 2017b), our continual learning algorithm builds on inducing-point methods (Titsias, 2009; Hensman et al., 2013; 2015).”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, methodology): “Then, based on the above we can jointly apply stochastic variational inference (Hensman et al., 2013) to maximise the ELBO over (θ, µu1 , Lu1) and optionally over the inducing inputs Z1.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “We consider the flight delay data set (Hensman et al., 2013; Deisenroth and Ng, 2015; Lakshminarayanan et al., 2016), a large scale regression benchmark with several published results.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “[22] James Hensman, Nicolò Fusi, and Neil D Lawrence.”
Stochastic Variational Deep Kernel Learning (2016, methodology): “In particular, a similar variational lower bound as in Eq (3) was proposed in [24, 10] for a sparse GP, which were extended to non-conjugate likelihoods, with the intractable integrals estimated using Gaussian quadrature as in the KLSP-GP [11] or univariate Gaussian samples as in the SAVI-GP [6].”
Deep Neural Networks as Gaussian Processes (2017, positive): “While we were able to run experiments for somewhat large datasets (sizes of 50k), we intend to look into scalability for larger learning tasks, possibly harnessing recent progress in scalable GPs (Qui˜nonero-Candela & Rasmussen (2005); Hensman et al. (2013)).”
Hidden physics models: Machine learning of nonlinear partial differential equations (2017, methodology): “However, ideas such as recursive Kalman updates [69], variational inference [55], and parametric Gaussian processes [56] can be used to address this limitation.”
Scalable Variational Gaussian Process Classification (2014, methodology): “To obtain the popular FITC method (in the case of Gaussian likelihood), a factorization is enforced: | ≈ | To get a variational approximation, the following inequality is used Substituting this bound on the conditional into the standard expression p ( y ) = (cid:82) p ( y | u ) p ( u )d u gives a…”
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) (2015, methodology): “Inducing point methods (Snelson and Ghahramani, 2006; Hensman et al., 2013; Qui˜nonero-Candela and Rasmussen, 2005; Seeger, 2005; Smola and Bartlett, 2001; Silverman, 1985) have been introduced to scale up Gaussian processes to larger data-sizes.”
Streaming Sparse Gaussian Process Approximations (2017, background): “Stochastic optimisation methods that employ mini-batches of training data can be used to further reduce computational costs [4, 5, 6, 7], allowing GPs to be scaled to datasets comprising millions of data points.”
Thoughts on Massively Scalable Gaussian Processes (2015, methodology): “Indeed, the methodology in MSGP and BDGP are complementary and could be combined for a particularly scalable Gaussian process framework.”
Automated Machine Learning on Big Data using Stochastic Algorithm Tuning (2014, methodology): “Methods in this family include stochastic optimisation [9], stochastic gradient Langevin dynamics [10] and stochastic variational inference [11, 22].”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, methodology): “We follow [14] and specify the variational approximation as a combination of the conditional GP prior and a variational distribution over the inducing function values, independent across output dimensions”
GPflow: A Gaussian Process Library using TensorFlow (2016, methodology): “To this end we support ‘variationally sparse’ methods which ensure that the approximation is scalable and close (Opper and Archambeau, 2009), SGPR (Titsias, 2009), SVGP (Hensman et al., 2013, 2015b) and SGPMC (Hensman et al., 2015a).”
Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations (2017, methodology): “In future work we plan to design more computationally efficient algorithms by exploring ideas including recursive Kalman updates [11] and variational inference [13].”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “[13, 14] introduced stochastic variational inference for improving scalability, where inducing points are regarded as latent variables and marginalized out in the model inference.”