Scalable Variational Gaussian Process Classification

By James Hensman, Alexander G. de G. Matthews, Zoubin Ghahramani. 2014

732 citations recorded by the source corpus.

Gaussian process classification is a popular method with a number of appealing properties. We show how to scale the model within a variational inducing point framework, outperforming the state of the art on benchmark datasets. Importantly, the variational formulation can be exploited to allow classification in problems with millions of data points, as we demonstrate in experiments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Probabilistic & Bayesian Methods

Citations in the corpus

GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “One immediate option is to build a tree and learn inducing inputs only for the novel classes seen at each new session similarly to the learning applied for base classes with VI.”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “We take a step in this direction and propose Variational Auto-Regressive Gaussian Processes (VAR-GPs) for continual learning.”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, methodology): “While in a static task maximising directly the ELBO of the sparse GP tends to give good performance (Titsias, 2009; Hensman et al., 2015), this does not need to be the case for continual learning where the feature vector undergoes changes as more tasks are encountered.”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “For the sparse version we used a Sparse Variational GP (Hensman et al., 2015) with the same kernel and with 300 inducing points, trained via minibatch stochastic gradient descent (Matthews et al., 2017).”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “[23] James Hensman, Alexander Matthews, and Zoubin Ghahramani.”
Stochastic Variational Deep Kernel Learning (2016, methodology): “In particular, a similar variational lower bound as in Eq (7) was proposed in [25, 11] for a sparse GP, which were extended to non-conjugate likelihoods, with the intractable integrals estimated using Gaussian quadrature as in the KLSP-GP [12] or univariate Gaussian samples as in the SAVI-GP [7].”
Streaming Sparse Gaussian Process Approximations (2017, background): “Stochastic optimisation methods that employ mini-batches of training data can be used to further reduce computational costs [4, 5, 6, 7], allowing GPs to be scaled to datasets comprising millions of data points.”
GPflow: A Gaussian Process Library using TensorFlow (2016, methodology): “As a scenario, we studied training a multiclass GP classiﬁer on MNIST using stochastic variational inference (Hensman et al., 2015b,a) 2 .”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, methodology): “To check the behavior of the proposed method qualitatively, we first turned to the Banana dataset 2 [14] for binary classification with 400 two-dimensional examples.”
Continual Multi-task Gaussian Processes (2019, methodology): “From a practical point of view, when t = 0 in the expression above, that is, the ﬁrst time step, we train the model using the bound in Hensman et al. (2015) in order to set φ new .”
Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation (2018, methodology): “Gaussian process classiﬁcation Hensman and Matthews Pólya-Gamma data augmentation Polson, Scott, and”
Parametric Gaussian process regression for big data (2017, unknown): “Thus, similar to [16] and [31], this observation paves the way for generalizing the regression framework outlined above to solve large scale classification problems using parametric Gaussian processes.”
Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction (2017, methodology): “To reduce this computational complexity, we develop our learning algorithm based on the sparse variational approach [41, 42, 43, 44].”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, background): “In particular, building on recent work on pseudo-point sparse approximations [Titsias, 2009, Hensman et al., 2015, Matthews et al., 2016, Bui et al., 2017b] and streaming approximations [Csató and Opper, 2002, Bui et al., 2017a], we develop a streaming variational approximation that approximates…”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “For the ﬁrst issue, the stochastic GPC derives the model evidence expressed as the integration over an one-dimensional Gaussian distribution, which can be adequately calculated using Gaussian-Hermite quadrature [234], [238].”
Non-Parametric Calibration for Classification (2019, methodology): “We extend an approach by Hensman et al. (2015) to our choice of likelihood.”
A Scalable Hierarchical Gaussian Process Classifier (2019, methodology): “Different values for T0 are tested: 0 (corresponding to SVGP), 3, 6, 9, 12, and 15.”
Random Feature Expansions for Deep Gaussian Processes (2016, methodology): “The results are also competitive (and sometimes superior) to those obtained by the variational GP ( VAR - GP ) in Hensman et al. (2015b).”