A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

By Thang D. Bui, Josiah Yan, Richard E. Turner. 2016

25 citations recorded by the source corpus.

Gaussian processes (GPs) are flexible distributions over functions that enable high-level assumptions about unknown functions to be encoded in a parsimonious, flexible and general way. Although elegant, the application of GPs is limited by computational and analytical intractabilities that arise when data are sufficiently numerous or when employing non-Gaussian models. Consequently, a wealth of GP approximation schemes have been developed over the last 15 years to address these key limitations. Many of these schemes employ a small set of pseudo data points to summarise the actual data. In this paper we develop a new pseudo-point approximation framework using Power Expectation Propagation (Power EP) that unifies a large number of these pseudo-point approximations. Unlike much of the previous venerable work in this area, the new framework is built on standard methods for approximate inference (variational free-energy, EP and power EP methods) rather than employing approximations to the probabilistic generative model itself. In this way all of approximation is performed at `inference time' rather than at `modelling time' resolving awkward philosophical and empirical questions that trouble previous approaches. Crucially, we demonstrate that the new framework includes new pseudo-point approximation methods that outperform current approaches on regression, classification and state space modelling tasks.

Subfields: Artificial Intelligence, Computer Networks and Communications, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Control and Systems Engineering

Citations in the corpus

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “Goodfellow et al. [20, §7] suggest that shallow RBF networks, although not making classiﬁers that generalise well, are immune to these adversarial examples.”
Deep Gaussian Processes for Regression using Approximate Expectation Propagation (2016, methodology): “The method used here, Fully Independent Training Conditional (FITC), can be interpreted as a member of the first category (Bui et al., 2016).”
Understanding Probabilistic Sparse Gaussian Process Approximations (2016, methodology): “The latest interesting development came with the connection that FITC can be obtained by approximating the GP posterior using Expectation Propagation (EP) [12, 13, 14].”
Variational Implicit Processes (2018, background): “| Therefore sparse GP approximation techniques [13, 35, 84, 85] are applicable, but we leave them to future work and consider an alternative approach that is related to random feature approximations of GPs [3, 27, 29, 50, 74].”
Advances in Variational Inference (2017, background): “Sparse inference can be either interpreted as a modeling choice or as an inference scheme [24].”