Thoughts on Massively Scalable Gaussian Processes

By Andrew Gordon Wilson, Christoph Dann, Hannes Nickisch. 2015

116 citations recorded by the source corpus.

We introduce a framework and early results for massively scalable Gaussian processes (MSGP), significantly extending the KISS-GP approach of Wilson and Nickisch (2015). The MSGP framework enables the use of Gaussian processes (GPs) on billions of datapoints, without requiring distributed inference, or severe assumptions. In particular, MSGP reduces the standard $O(n^3)$ complexity of GP learning and inference to $O(n)$, and the standard $O(n^2)$ complexity per test point prediction to $O(1)$. MSGP involves 1) decomposing covariance matrices as Kronecker products of Toeplitz matrices approximated by circulant matrices. This multi-level circulant approximation allows one to unify the orthogonal computational benefits of fast Kronecker and Toeplitz approaches, and is significantly faster than either approach in isolation; 2) local kernel interpolation and inducing points to allow for arbitrarily located data inputs, and $O(1)$ test time predictions; 3) exploiting block-Toeplitz Toeplitz-block structure (BTTB), which enables fast inference and learning when multidimensional Kronecker structure is not present; and 4) projections of the input space to flexibly model correlated inputs and high dimensional data. The ability to handle many ($m \approx n$) inducing points allows for near-exact accuracy and large scale kernel learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Scaling & Learning Theory, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

In Probabilistic & Bayesian Methods

Citations in the corpus

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “These can be fewer in number than the number of data points and optimised in position [58, 55, 4, 7] or be large in number but placed in a computationally e�cient structure [62, 60].”
Stochastic Variational Deep Kernel Learning (2016, methodology): “Without requiring any grid structure in X , KZ,Z decomposes into a Kronecker product of Toeplitz matrices, which can be approximated by circulant matrices [29].”
State Space Gaussian Processes with Non-Gaussian Likelihood (2018, methodology): “Since the matrix exponential ψ : s 7→ e is smooth, its evaluation can be accurately approximated by convolution interpolation (Keys, 1981) as done for the covariance functions in the KISS-GP framework (Wilson & Nickisch, 2015; Wilson et al., 2015).”
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration (2018, methodology): “In KISS-GP [50, 51], the matrix KUU is also chosen to have algebraic structure, such as Kronecker or Toeplitz structure, which further accelerates MVMs.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “To address this issue, one could use dimensionality reduction or manifold learning to map the inducing points into a p-dimensional (p d) latent space [98], or more interestingly, one can use the hierarchical structure of NNs to extract the latent low-dimensional feature space [34], [99].”
Fast and Scalable Gaussian Process Modeling with Applications to Astronomical Time Series (2017, methodology): “Many other approximate methods for scaling GP inference exist (see, e.g., Wilson et al. 2015, and references therein), and we make no attempt to make our discussion exhaustive.”
Learning Scalable Deep Kernels with Recurrent Structure (2016, methodology): “Following Wilson et al. (2015), we performed a generic scalability analysis of the MSGP-LSTM model on the car sensors data.”
A Scalable Hierarchical Gaussian Process Classifier (2019, methodology): “For applying this method to datasets of higher dimensional inputs, the data inputs must first be projected into a lower dimensional using linear projections as in [68] or non-linear projections with deep neural networks as in the deep kernel learning (DKL) method [18].”