Gaussian Process Kernels for Pattern Discovery and Extrapolation

By Andrew Gordon Wilson, Ryan P. Adams. 2013

682 citations recorded by the source corpus.

Gaussian processes are rich distributions over functions, which provide a Bayesian nonparametric approach to smoothing and interpolation. We introduce simple closed form kernels that can be used with Gaussian processes to discover patterns and enable extrapolation. These kernels are derived by modelling a spectral density - the Fourier transform of a kernel - with a Gaussian mixture. The proposed kernels support a broad class of stationary covariances, but Gaussian process inference remains simple and analytic. We demonstrate the proposed kernels by discovering patterns and performing long range extrapolation on synthetic examples, as well as atmospheric CO2 trends and airline passenger data. We also show that it is possible to reconstruct several popular standard covariances within our framework.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep Kernel Learning (2015, methodology): “For added flexibility, we also propose to use spectral mixture base kernels (Wilson and Adams, 2013):”
Stochastic Variational Deep Kernel Learning (2016, background): “Indeed many recent approaches have advocated building expressive kernel functions [e.g., 23, 10, 27, 26, 18, 32], and emerging research in this direction takes inspiration from deep learning models [e.g., 29, 6, 3].”
Manifold Gaussian Processes for regression (2014, methodology): “The ﬁrst approach combines multiple standard covariance functions to form a new covariance function (Rasmussen and Williams, 2006; Wilson and Adams, 2013; Duve-naud et al., 2013).”
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) (2015, background): “More expressive kernels, such as the spectral mixture kernel (Wilson and Adams, 2013), will require more inducing (interpolation) points for a good approximation, due to their quasi-periodic nature.”
A la Carte - Learning Fast Kernels (2014, background): “In other words, rather than choosing a spherically symmetric representation ρ(ω) as typical for (2), Wilson and Adams [2013] pick a mixture of Gaussians with mean frequency μq and variance Σq that satisfy the symmetry condition ρ(ω) = ρ(−ω) but not rotation invariance.”
Automatic model construction with Gaussian processes (2014, background): “, 2010), spectral mixture kernels (Wilson and Adams, 2013) (spectral kernels), and trend-cyclical-irregular models (e.”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “, 2016) with a spectral mixture base kernel (Wilson and Adams, 2013), SGD trained”
Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (2020, methodology): “For this reason, a variety of other kernels has been proposed in the literature: Radial Basis Function kernel (RBF), Matérn kernel, Cosine Similarity kernel (CosSim), and the spectral mixture kernel (Wilson and Adams, 2013).”
The Human Kernel (2015, methodology): “For this approach, we choose the recent spectral mixture kernels of Wilson and Adams [14], which can model a wide range of stationary covariances, and are intended to help automate kernel selection.”
Large-scale Online Kernel Learning with Random Feature Reparameterization (2017, background): “Moreover, in the light of our proposed framework, a class of reparameterized random features for other kernels (e.g., spectral mixture kernels [Wilson and Adams, 2013] with both parametric and nonparametric [Oliva et al., 2015] versions) can also be derived as long as an appropriate mapping can be…”
Residential probabilistic load forecasting: A method using Gaussian process designed for electric load data (2018, methodology): “Each GP is modeling a specific trend using a specific kernel [52].”
Bayesian Nonparametric Kernel-Learning (2015, background): “[27] considers kernels of the form in (4), with parameters chosen heuristically; [26] mentions the possibility of putting a DP prior model on parameters, but does so without a sampling algorithm and empirical details.”
Sparse Multi-Output Gaussian Processes for Medical Time Series Prediction (2017, methodology): “For each basis kernel that was estimated, the characteristic period is 1/μq and the length scale is 1/2πvq (Wilson and Adams, 2013).”
Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs (2015, background): “As an example covariance function of this form consider the spectral mixture (SM) covariance function (Lindgren, 2012; Wilson & Adams, 2013).”
Gaussian Processes for Independence Tests with Non-iid Data in Causal Inference (2015, background): “(5) Our new GP formulation for structured inputs and outputs, as introduced in Section 4.4, opens up the possibility of conditional independence and causal inference with structured data such as text, images, and anything else on which a Mercer kernel can be deﬁned.”