Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition

By Pavel Izmailov, Alexander Novikov, Dmitry Kropotov. 2017

69 citations recorded by the source corpus.

We propose a method (TT-GP) for approximate inference in Gaussian Process (GP) models. We build on previous scalable GP research including stochastic variational inference based on inducing inputs, kernel interpolation, and structure exploiting algebra. The key idea of our method is to use Tensor Train decomposition for variational parameters, which allows us to train GPs with billions of inducing inputs and achieve state-of-the-art results on several benchmarks. Further, our approach allows for training kernels based on deep neural networks without any modifications to the underlying GP model. A neural network learns a multidimensional embedding for the data, which is used by the GP to make the final prediction. We train GP and neural network parameters end-to-end without pretraining, through maximization of GP marginal likelihood. We show the efficiency of the proposed approach on several regression and classification benchmark datasets including MNIST, CIFAR-10, and Airline.

Subfields: Artificial Intelligence, Computational Mathematics, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Guodong Zhang
2. Roger Baker Grosse
3. Jiaxin Shi
4. Ethan Fetaya
5. Florian Wenzel
6. Gal Chechik
7. Aviv Navon
8. Théo Galy-Fajou

In Computational Mathematics

1. Roger Baker Grosse
2. Jiaxin Shi
3. Guodong Zhang

In Statistical and Nonlinear Physics

1. Guodong Zhang
2. Roger Baker Grosse
3. Manfred Opper
4. Gal Chechik
5. Jiaxin Shi
6. Ethan Fetaya
7. Matthias W. Seeger
8. James Martens

Citations in the corpus

• GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “Izmailov et al. (2018) used tensor train decomposition for variational parameters that allowed increasing dramatically the number of inducing points.”
• Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation (2019, background): “Izmailov et al. (2018) use tensor train decomposition to use high numbers of inducing points but do not provide efficient closed-form updates.”
• Functional Variational Bayesian Neural Networks (2019, methodology): “…Snelson & Ghahramani, 2006; Titsias, 2009; Hensman et al., 2013; 2015; Krauth et al., 2016) still suffer for very large dataset, while random feature methods (Rahimi & Recht, 2008; Le et al., 2013) and KISS-GP (Wilson & Nickisch, 2015; Izmailov et al., 2017) must be hand-tailored to a given kernel.”