Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs

By Yarin Gal, Richard E. Turner. 2015

82 citations recorded by the source corpus.

Standard sparse pseudo-input approximations to the Gaussian process (GP) cannot handle complex functions well. Sparse spectrum alternatives attempt to answer this but are known to over-fit. We suggest the use of variational inference for the sparse spectrum approximation to avoid both issues. We model the covariance function with a finite Fourier series approximation and treat it as a random variable. The random covariance function has a posterior, on which a variational distribution is placed. The variational distribution transforms the random covariance function to fit the data. We study the properties of our approximate inference, compare it to alternative ones, and extend it to the distributed and stochastic domains. Our approximation captures complex functions better than standard approaches and avoids over-fitting.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference (2015, background): “Gal and Turner [11] have shown that the Gaussian process can be approximated by defining ω = {M̂1, M̂2} to be an approximating distribution over the spectral frequencies and their coefficients in a Fourier decomposition of our function:”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, result): “the missing segments. However the GP’s uncertainty is larger and over-estimates its 95% conﬁdence interval in capturing the true function. The observed uncertainty in MC dropout is similar to that of [32] with the model over-conﬁdent in its predictions. Note that this is often observed with variatio”
Variational Implicit Processes (2018, unknown): “| Therefore sparse GP approximation techniques [13, 35, 84, 85] are applicable, but we leave them to future work and consider an alternative approach that is related to random feature approximations of GPs [3, 27, 29, 50, 74].”
Blitzkriging: Kronecker-structured Stochastic Gaussian Processes (2015, background): “Titsias [11] and Gal and Turner [12] address this by representing the uncertainty in the inducing and spectral points, respectively, with a variational distribution.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “Alternatively, the sparse spectrum GP (SSGP) [61] and its variational variants [62]–[64] elegantly address this issue by reconstructing the Bayesian linear model from the spectral representation (Fourier features), resulting in the stationary kernel”
Dropout as a Bayesian Approximation : Insights and Applications (2015, methodology): “It has been applied in both the supervised and unsupervised domains , for both regression and classification tasks (Rasmussen & Williams, 2006; Titsias & Lawrence, 2010; Gal et al., 2015).”