Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)

By Andrew Gordon Wilson, Hannes Nickisch. 2015

579 citations recorded by the source corpus.

We introduce a new structured kernel interpolation (SKI) framework, which generalises and unifies inducing point methods for scalable Gaussian processes (GPs). SKI methods produce kernel approximations for fast computations through kernel interpolation. The SKI framework clarifies how the quality of an inducing point approach depends on the number of inducing (aka interpolation) points, interpolation strategy, and GP covariance kernel. SKI also provides a mechanism to create new scalable kernel methods, through choosing different kernel interpolation strategies. Using SKI, with local cubic kernel interpolation, we introduce KISSGP, which is 1) more scalable than inducing point alternatives, 2) naturally enables Kronecker and Toeplitz algebra for substantial additional gains in scalability, without requiring any grid data, and 3) can be used for fast and expressive kernel learning. KISS-GP costs O(n) time and storage for GP inference. We evaluate KISS-GP for kernel matrix approximation, kernel learning, and natural sound modelling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “This expectation can be computed in closed form for GPs.”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, background): “The recent advances to scale GPs to highdimensional input include Kronecker product decomposition that exploits the spatial structure [1, 4, 5], convolutional kernels [6] and other algorithmic and software developments [7].”
Neural Processes (2018, methodology): “To overcome this computational complexity DKL replace a standard GP with a kernel approximation given by a KISS GP (Wilson & Nickisch, 2015), while prototypical networks (Snell et al., 2017) are introduced as a more light-weight version of MNs that also scale with O ( n + m ) .”
Deep Kernel Learning (2015, methodology): “We further build on the ideas in KISS-GP (Wilson and Nickisch, 2015) and extensions (Wilson et al., 2015), so that our deep kernel learning model can scale linearly with the number of training instances n, instead of O(n3) as is standard with Gaussian processes, while retaining a fully…”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “These can be fewer in number than the number of data points and optimised in position [58, 55, 4, 7] or be large in number but placed in a computationally e�cient structure [62, 60].”
Stochastic Variational Deep Kernel Learning (2016, background): “We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.”
Thoughts on Massively Scalable Gaussian Processes (2015, methodology): “While Wilson and Nickisch (2015) propose fast O ( n ) inference and learning, test time predictions are the same as for a standard GP – namely, O ( n ) for the predictive mean and O ( n 2 ) for the predictive variance per single test point x ∗ .”
Scalable Log Determinants for Gaussian Process Kernel Learning (2017, methodology): “However, evaluating the marginal likelihood and its derivatives, for kernel learning, has followed a scaled eigenvalue approach [12, 13] instead of iterative MVM approaches.”
Functional Variational Bayesian Neural Networks (2019, methodology): “…Snelson & Ghahramani, 2006; Titsias, 2009; Hensman et al., 2013; 2015; Krauth et al., 2016) still suffer for very large dataset, while random feature methods (Rahimi & Recht, 2008; Le et al., 2013) and KISS-GP (Wilson & Nickisch, 2015; Izmailov et al., 2017) must be hand-tailored to a given kernel.”
State Space Gaussian Processes with Non-Gaussian Likelihood (2018, methodology): “A second class of covariance interpolation methods, KISS-GP (Wilson & Nickisch, 2015; Wilson et al., 2015), are based on grids of inducing points.”
Machine-learning-guided directed evolution for protein engineering (2018, background): “The run time for exact GP regression scales as the cube of the number of training examples, which makes it unsuitable for large ( > 10 3 ) datasets, but there are now fast and accurate approximations 41,42 .”
Machine learning in protein engineering (2018, background): “The run-time for exact GP regression scales as the number of training examples cubed, making it unsuitable for large ( > 10 3 ) datasets, but there are now fast and accurate approximations (Wilson & Nickisch, 2015).”
Adaptive Deep Kernel Learning (2019, methodology): “Some approximations of the kernel [21, 22] are thus needed for the scalability of the DKL method.”
A Scalable Hierarchical Gaussian Process Classifier (2019, background): “In [65], Wilson and Nickisch introduce a unifying framework for inducing point methods, called structured kernel interpolation (SKI).”
A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification (2016, methodology): “Wilson and Nickisch [25] showed that when the kernel is locally smooth (under the resolution of u), cubic interpolation results in accurate approximation.”
Stochastic Chebyshev Gradient Descent for Spectral Optimization (2018, methodology): “For handling large-scale datasets, [32] proposed the structured kernel interpolation framework assuming θ = [ θ i ] ∈ R 3 and where W ∈ R d × r is some sparse matrix and K ∈ R r × r is a dense kernel with r d .”
Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition (2017, background): “…case the interpolation weights wi can be represented as a Kronecker product over dimensions wi = w 1 i ⊗w2i ⊗. . .⊗wDi , wi ∈ Rmi ∀i = 1, 2, . . . , D. Wilson and Nickisch (2015) combine these ideas with SOR (Silverman (1985)) in the KISS-GP method yielding O(n+Dm1+1/D) computational complexity.”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “• Scalable BO that utilizes modern numerical linear algebra from GPyTorch (Gardner et al., 2018), including sparse interpolation and structure exploiting algebra (Wilson & Nickisch, 2015), and fast predictive distributions and sampling (Pleiss et al., 2018).”
Gaussian process regression for forecasting battery state of health (2017, background): “…schemes have been proposed in the literature, based on approximating input data with pseudo points [24], exploiting grid structure [25] and exploiting the recursivity of the estimation problem in online settings [22, 26, 27]; and these could be used if larger numbers of…”