Variational Auto-encoded Deep Gaussian Processes

By Zhenwen Dai, Andreas C. Damianou, Javier I. González, Neil D. Lawrence. 2015

138 citations recorded by the source corpus.

We develop a scalable deep non-parametric generative model by augmenting deep Gaussian processes with a recognition model. Inference is performed in a novel scalable variational framework where the variational posterior distributions are reparametrized through a multilayer perceptron. The key aspect of this reformulation is that it prevents the proliferation of variational parameters which otherwise grow linearly in proportion to the sample size. We derive a new formulation of the variational lower bound that allows us to distribute most of the computation in a way that enables to handle datasets of the size of mainstream deep learning tasks. We show the efficacy of the method on a variety of challenges including deep unsupervised learning and deep Bayesian optimization.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Stochastic Variational Deep Kernel Learning (2016, background): “, [18, 10]), and some of the most recent work in this area applied variational auto-encoders [14] for coupled variational updates (aka back constraints) [4, 2].”
Deep Gaussian Processes for Regression using Approximate Expectation Propagation (2016, background): “However, both approaches of (Dai et al., 2016) and (Hensman & Lawrence, 2014) have not been fully evaluated on benchmark supervised learning tasks or on medium to large scale datasets, nor compared to alternative deep models.”
Random Feature Expansions for Deep Gaussian Processes (2016, methodology): “Notable examples include the extension of inducing point approximations (Hensman & Lawrence, 2014; Dai et al., 2016) and Expectation Propagation (Bui et al., 2016). Sequential inference for training DGPs has also been investigated in Wang et al. (2016). A recent example of a DGP “natively” formulate”
Advances in Variational Inference (2017, background): “Instead of estimating these latent variables separately, however, [34] proposes to estimate these latent variables as functions of inference networks, allowing DGPs to scale to bigger data-”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, methodology): “This approach is followed in Dai et al. (2016) and Cutajar et al. (2017).”
When Gaussian Process Meets Big Data: A Review of Scalable GPs (2018, background): “More elegantly, inspired by deep learning, the DGP [189] and its variants [196]–[200], which employ the hierarchical and functional composite”
MXFusion: A Modular Deep Probabilistic Programming Library (2018, methodology): “It is straightforward to construct a probabilistic model consists of multiple probabilistic modules such as deep Gaussian processes (GP) (Damianou & Lawrence, 2013; Dai et al., 2016; Salimbeni & Deisenroth, 2017).”