GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

By Jacob V. Gardner, Geoff Pleiss, David S. Bindel, Kilian Q. Weinberger, Andrew Gordon Wilson. 2018

1,460 citations recorded by the source corpus.

Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from $O(n^3)$ to $O(n^2)$. Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “[16] Gardner, J.”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, background): “The recent advances to scale GPs to highdimensional input include Kronecker product decomposition that exploits the spatial structure [1, 4, 5], convolutional kernels [6] and other algorithmic and software developments [7].”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “, 2016b) which use the same networks used in the previous experiment (minus the top layer) as their feature extractors, following the implementation provided by GPyTorch (Gardner et al., 2018).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “We use the GPyTorch package (Gardner et al., 2018) to train the models.”
Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes (2020, methodology): “After task recognition, the corresponding dynamics model parameters are updated via conjugate gradient [15].”
Bayesian reaction optimization as a tool for chemical synthesis (2021, methodology): “Over the course of our studies we found that a Gaussian process model 29 with the Matérn52 kernel offered superior optimization performance over a random forest model 36 in terms of mean loss, outcome variance and worst-case loss (Extended Data Table 1).”
Scalable Global Optimization via Local Bayesian Optimization (2019, methodology): “To address computational issues, we use GPyTorch [16] for scalable GP regression.”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, methodology): “Moreover, by further using the relaxed ELBO (17) or grid-inducing points in (20), the TensorFlowbased GPflow library [31] and the PyTorch-based GPyTorch library [158] have been developed to exploit the usage of GPU hardwares.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “For Gaussian processes, there have been signiﬁcant developments in libraries (Rasmussen and Nickisch, 2010; GPy, 2012; Vanhatalo et al., 2013; Matthews et al., 2017; Al-Shedivat et al., 2017; Gardner et al., 2018).”
Deep Factors with Gaussian Processes for Forecasting (2018, methodology): “The model is implemented in MXNet Gluon [20] with a RBF kernel [21] using the mxnet.”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “To compute these eigenvalues, we used a GPU-enabled Lanczos method in GPyTorch (Gardner et al., 2018) on a pre-trained PreResNet164.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “, 2017) and GPyTorch (Gardner et al., 2018) are two GP libraries built on top of Tensorflow and PyTorch, respectively, and thus can be used to implement surrogate models for BO.”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “Instead of sequentially evaluating an acquisition function at a number of candidate sets x 1 , . . . , x b , where x k ∈ R q × d for each k , B O T ORCH evaluates a batched tensor X ∈ R b × q × d .”
Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels (2020, methodology): “We integrated our algorithm into this framework using PyTorch and GPyTorch (Gardner et al., 2018).”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, methodology): “We employ an efficient implementation of the Lanczos method for determining eigenvalues provided in GPyTorch, allowing for the rapid computation of approximate eigenvalue, eigenvector pairs (Gardner et al., 2018).”