Random Feature Expansions for Deep Gaussian Processes

By Kurt Cutajar, Edwin V. Bonilla, Pietro Michiardi, Maurizio Filippone. 2016

153 citations recorded by the source corpus.

The composition of multiple Gaussian Processes as a Deep Gaussian Process (DGP) enables a deep probabilistic nonparametric approach to flexibly tackle complex machine learning problems with sound quantification of uncertainty. Existing inference approaches for DGP models have limited scalability and are notoriously cumbersome to construct. In this work we introduce a novel formulation of DGPs based on random feature expansions that we train using stochastic variational inference. This yields a practical learning framework which significantly advances the state-of-the-art in inference for DGPs, and enables accurate quantification of uncertainty. We extensively showcase the scalability and performance of our proposal on several datasets with up to 8 million observations, and various DGP architectures with up to 30 hidden layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “People have built more expressive functions by automatically combining kernels together [14] as well as by composing GPs together [13, 11, 15].”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, methodology): “Approximations are given in the form of SparseGaussianProcess for inducing points (leading to Salimbeni and Deisenroth (2017)) and RandomFourierFeatures for finite trigonometric basis function approximations (used by Cutajar et al. (2017)).”
Functional Variational Bayesian Neural Networks (2019, background): “Deep Gaussian Processes (DGP) (Cutajar et al., 2017; Salimbeni & Deisenroth, 2017) have close connections to BNNs due to similar deep structures.”
Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition (2017, methodology): “…our method with standard RBF covariance functions we used a range of classification and regression tasks from UCI and LIBSVM archives and the Airline dataset, that is popular for testing scalable GP models (Hensman et al. (2013), Hensman et al. (2015), Wilson et al. (2016b), Cutajar et al. (2016)).”
Variational Implicit Processes (2018, background): “These include: (1) deep GPs [11, 17, 18], where compositions of GP priors are proposed to represent prior over compositional functions; (2) the search and design of kernels for accurate and efﬁcient learning [3, 5, 24, 79, 86, 89], and (3) deep kernel learning that uses deep neural net features as…”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “More elegantly, inspired by deep learning, the deep GP (DGP) [201] and its variants [202]–[205], which employ the hierarchical and functional composite”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, methodology): “In Cutajar et al. (2017) a fully factorized variational approximation is used for the spectral components.”