The Variational Gaussian Process

By Dustin Tran, Rajesh Ranganath, David M. Blei. 2015

194 citations recorded by the source corpus.

Variational inference is a powerful tool for approximate inference, and it has been recently applied for representation learning with deep generative models. We develop the variational Gaussian process (VGP), a Bayesian nonparametric variational family, which adapts its shape to match complex posterior distributions. The VGP generates approximate posterior samples by generating latent inputs and warping them through random non-linear mappings; the distribution over random mappings is learned during inference, enabling the transformed outputs to adapt to varying complexity. We prove a universal approximation theorem for the VGP, demonstrating its representative power for learning any model. For inference we present a variational objective inspired by auto-encoders and perform black box inference over a wide class of models. The VGP achieves new state-of-the-art results for unsupervised learning, inferring models such as the deep latent Gaussian model and the recently proposed DRAW.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “− log p ( x ) Results with autoregressive components DRAW+VGP [53] 79 .”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “, 1995) or variants thereof (Kingma & Welling, 2013; Rezende et al., 2014; de Freitas et al., 2001; Gregor et al., 2015b;a; Tran et al., 2015; Kingma et al., 2016) trained with approximate inference.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “, 2016) [4]: (Tran et al., 2015) Model VLB log p(x) ⇡ Convolutional VAE + HVI [1] -83.”
Density estimation using Real NVP (2016, background): “Still, the approximation in the inference process limits its ability to learn high dimensional deep representations, motivating recent work in improving approximate inference [42, 48, 55, 63, 10, 59, 34].”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, background): “, it represents about half of the gap between a state-of-theart permutation-invariant model [28] and one which exploits structure [30].”
Variational Lossy Autoencoder (2016, neutral): “| || | Many previous works in Variational Inference have designed ﬂexible approximate posteriors to better approximate true posterior (Salimans et al., 2014; Rezende & Mo-hamed, 2015; Tran et al., 2015; Kingma et al., 2016).”
Auxiliary Deep Generative Models (2016, background): “, 2016) and variational Gaussian process VAE (Tran et al., 2015).”
Ladder Variational Autoencoders (2016, methodology): “Further the LVAE is orthogonal to other methods for improving the inference distribution such as Normalizing ﬂows [17], Variational Gaussian Process [20] or Auxiliary Deep generative models [12] and combining with these might provide further improvements.”
Evidential Deep Learning to Quantify Classification Uncertainty (2018, neutral): “Hence they are universal predictors [31].”
Variational Inference: A Review for Statisticians (2016, background): “…covariance; Giordano, Broderick, and Jordan (2015) post-processed the mean-field parameters to correct for underestimating the variance; and Ranganath, Tran, and Blei (2016) embedded the mean-field parameters themselves in a hierarchical model to induce variational dependencies between…”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “…u, (cid:126)z, (cid:126)x,(cid:126)a,(cid:126)r ) using Variational Auto-Encoders (VAEs) (Kingma and Welling, 2013), which can represent a very large class of distributions with latent variables and are easily implemented by solving an optimization problem (Tran et al., 2015; Louizos et al., 2017).”
Hierarchical Variational Models (2015, methodology): “In other work, we use Gaussian processes (Tran et al., 2016b).”
Discrete Variational Autoencoders (2016, background): “10 Variational Gaussian process [29] -81.”
VAE with a VampPrior (2017, background): “Various extensions to deep generative models have been proposed that aim to enrich the variational posterior [10, 29, 32, 35, 39, 40].”
Structured Inference Networks for Nonlinear State Space Models (2016, background): “However, one could augment our proposed inference network with recent innovations that improve the variational approximation to allow for multi-modality (Rezende and Mohamed 2015; Tran, Ranganath, and Blei 2016).”
Sylvester Normalizing Flows for Variational Inference (2018, background): “Therefore, designing tractable and more expressive variational families is an important problem in variational inference (Nalisnick et al., 2016; Salimans et al., 2015; Tran et al., 2015).”
Improving Variational Auto-Encoders using Householder Flow (2016, methodology): “In our approach we need to store only T × M more parameters comparing to the vanilla VAE while, e.g. , the VAE+NICE and VAE+NF require M × ( M − 1)”