Doubly Stochastic Variational Inference for Deep Gaussian Processes

By Hugh Salimbeni, Marc Peter Deisenroth. 2017

472 citations recorded by the source corpus.

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to over-fitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm, which does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Conditional Neural Processes (2018, background): “Scaling issues with GPs have been addressed by sparse GPs (Snelson & Ghahramani, 2006), while overcoming the limited expressivity resulting from functional restrictions is the motivation for Deep GPs (Damianou & Lawrence, 2013; Salimbeni & Deisenroth, 2017).”
Structured Bayesian Gaussian process latent variable model: applications to data-driven dimensionality reduction and high-dimensional inversion (2018, background): “While imposing Gaussian or Laplace priors over the model weights [23] is attractive because of its mathematical tractability [24], one may still question how well one understands the distributions over functions they imply [25].”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “It takes an ensemble approach and aims to work out-of-the-box across a wide range of problems, a design choice that makes it relatively hard to extend.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “Deep Gaussian processes (DGPs) [136], [137], [138], [139], [140], [141], [142] are effective multi-layer decision making models that can accurately model the uncertainty.”
Probabilistic Recurrent State-Space Models (2018, methodology): “The inference scheme is inspired by doubly stochastic variational inference for deep GPs as presented in (Salimbeni & Deisenroth, 2017).”
MXFusion: A Modular Deep Probabilistic Programming Library (2018, unknown): “It is straightforward to construct a probabilistic model consists of multiple probabilistic modules such as deep Gaussian processes (GP) (Damianou & Lawrence, 2013; Dai et al., 2016; Salimbeni & Deisenroth, 2017).”
Deep convolutional Gaussian processes (2018, methodology): “alized by running k-means with M clusters on image patches from the training set. „e variational means m are initialised to zero. S are initialised to a tiny variance kernel prior 10 5 K ZZ following Salimbeni and Deisenroth (2017), except for the last layer where we use K ZZ. For models deeper than”
Functional Variational Bayesian Neural Networks (2019, background): “Lee et al. (2018) recently extended this result to deep BNNs. Deep Gaussian Processes (DGP) (Cutajar et al., 2017; Salimbeni & Deisenroth, 2017) have close connections to BNNs due to similar deep structures.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “…by research on priors and algorithms for Bayesian neural networks (e.g., Wen et al., 2018; Hafner et al., 2018), scaling up Gaussian processes (e.g., Salimbeni and Deisenroth, 2017; John and Hensman, 2018), and expressive distributions via invertible functions (e.g., Rezende and Mohamed, 2015).”