Hierarchical Variational Models

By Rajesh Ranganath, Dustin Tran, David M. Blei. 2015

365 citations recorded by the source corpus.

Black box variational inference allows researchers to easily prototype and evaluate an array of models. Recent advances allow such algorithms to scale to high dimensions. However, a central question remains: How to specify an expressive variational distribution that maintains efficient computation? To address this, we develop hierarchical variational models (HVMs). HVMs augment a variational approximation with a prior on its parameters, which allows it to capture complex structure for both discrete and continuous latent variables. The algorithm we develop is black box, can be used for any HVM, and has the same computational efficiency as the original approximation. We study HVMs on a variety of deep discrete latent variable models. HVMs generalize other expressive variational distributions and maintains higher fidelity to the posterior.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “R | Agakov and Barber [1], Salimans et al. [61], Ranganath et al. [57], Maaløe et al. [46] develop an auxiliary variable variational bound (1)), however, the density of the EIM q ( z x ) is intractable. where r ( | z, x ) is a variational distribution meant to model q ( | z, x ) , and the identity…”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “The inspiration for the bidirectional inference network of BIVA comes from the work on Auxiliary VAEs (AVAE) by [37, 31].”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “An alternative method for increasing the flexiblity of the variational inference, is the introduction of auxiliary latent variables (Salimans et al., 2014; Ranganath et al., 2015; Tran et al., 2015) and corresponding auxiliary inference models.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “In deep hierarchical VAEs [5, 9, 4, 42, 43], to increase the expressiveness of both the approximate posterior and prior, the latent variables are partitioned into disjoint groups, zzz = { zzz 1 ,zzz 1 , . . . ,zzz L } , where L is the number of groups.”
Auxiliary Deep Generative Models (2016, methodology): “The AVAE is performing better than the VAE with normalizing ﬂows (Rezende and Mohamed, 2015) and the importance weighted auto-encoder with 1 IW sample (Burda et al., 2015).”
Variational Inference: A Review for Statisticians (2016, background): “…covariance; Giordano, Broderick, and Jordan (2015) post-processed the mean-field parameters to correct for underestimating the variance; and Ranganath, Tran, and Blei (2016) embedded the mean-field parameters themselves in a hierarchical model to induce variational dependencies between…”
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks (2017, methodology): “There has been a lot of recent work on ways to improve the posterior approximation in latent variable models with normalizing flows (Rezende & Mohamed, 2015) and auxiliary random variables (Agakov & Barber, 2004; Salimans et al., 2013; Ranganath et al., 2015; Maaløe et al., 2016) being the most prom”
The Variational Gaussian Process (2015, methodology): “Various versions of this objective have been considered in the literature (Jaakkola & Jordan, 1998; Agakov & Barber, 2004), and it has been recently revisited by Salimans et al. (2015) and Ranganath et al. (2015b).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “Rank-1 BNNs’ variational approximation can be seen as a form of hierarchical variational model (Ranganath et al., 2016) similar to multiplicative normalizing flows, which posit an auxiliary distribution on the hidden units (Louizos & Welling, 2017).”
VAE with a VampPrior (2017, background): “Last but not least, it would be interesting to utilize a normalizing flow [32], the hierarchical variational inference [31], ladder networks [36] or adversarial training [28] within the VampPrior VAE.”
Automatic Differentiation Variational Inference (2016, background): “A class of hierarchical variational models (Ranganath et al., 2015) extend these complex distributions to discrete latent variable models.”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, methodology): “…variational inference methods with rich approximation families; the recent successful algorithms (e.g., Rezende & Mohamed, 2015b; Tran et al., 2015; Ranganath et al., 2015, to name only a few) have to handcraft special variational families to ensure the computational tractability of q η ( x ) and…”
Inference Suboptimality in Variational Autoencoders (2018, methodology): “Our choice of ﬂow is a combination of the Real NVP (Dinh et al., 2017) and auxiliary variables (Ranganath et al., 2016; Maaløe et al., 2016).”
Deep and Hierarchical Implicit Models (2017, methodology): “An indirect approach requires constructing a sufﬁciently broad class of functions with posterior expectation zero based on Stein’s method (Ranganath et al., 2016b).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “This boils down to the hierarchical variational method (Agakov & Barber, 2004; Ranganath et al., 2016) as a special case when one step of autoencoding transform is applied.”
The Blessings of Multiple Causes (2018, background): “A final example of a factor model is the deep exponential family (def) (Ranganath et al. 2015).”
Semi-Amortized Variational Autoencoders (2018, background): “However, recent advances in stochastic (Hoffman et al., 2013), black-box (Ranganath et al., 2014; 2016), and amortized (Mnih & Gregor, 2014; Kingma & Welling, 2014; Rezende et al., 2014) variational inference have made it possible to scale to large datasets and rich, non-conjugate models (see Blei…”
Structured Exploration via Hierarchical Variational Policy Networks (2018, background): “The latent variables λt introduce dependencies among the at, hence this policy is more flexible compared to standard fully factorized policies (Ranganath et al., 2015).”
Semi-Implicit Variational Inference (2018, methodology): “, 2012; Titsias & Lázaro-Gredilla, 2014), and constructing a hierarchical variational distribution (Ranganath et al., 2016; Tran et al., 2017).”