Understanding Probabilistic Sparse Gaussian Process Approximations

By Matthias Bauer, Mark van der Wilk, Carl E. Rasmussen. 2016

286 citations recorded by the source corpus.

Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “To summarise this means that we are optimising the following variational lower bound during training [23, §4]: Where q ( f x ) is our variational approximation to the distribution of f x and Z are the locations of our inducing points.”
Streaming Sparse Gaussian Process Approximations (2017, background): “Interestingly this appears to be related to the tendency for EP, in batch settings, to clump pseudo-inputs on top of one another [21].”
Online Sparse Multi-Output Gaussian Process Regression and Learning (2019, methodology): “The variational formulation of sparse approximation was proposed in [39]–[43] and shown in [44] to outperform the FITC approximation.”
Convolutional Gaussian Processes (2017, methodology): “Other methods, such as EP/FITC [14, 20, 21, 22], can be seen as approximate models that do not share this property, leading to behaviour that would not be expected from the model that is to be approximated [23].”
A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (2016, methodology): “Similarly, if the pseudo-input locations are optimised, the principled non-parametric model has suddenly acquired MD parameters and with them all of the concomitant issues of parametric models including overfitting and optimisation difficulties (Bauer et al., 2016).”
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation (2016, background): “The main approaches of this sort can be characterised in terms of two parallel frameworks that are described in the following sections.”
Statistical Model Aggregation via Parameter Matching (2019, unknown): “[1] previously showed that increasing number of inducing inputs tends to improve performance.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “Hence, the trace term is a regularizer guards against over-fitting, seeks to deliver a good inducing set, and always improves Fq with increasing m (see the theoretical analysis in [69] and [71]).”