Learnable Explicit Density for Continuous Latent Space and Variational Inference

By Chin-Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei, Laurent Charlin, Aaron C. Courville. 2017

28 citations recorded by the source corpus.

In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and inference. Second, we analyze the family of inverse autoregressive flows (inverse AF) and show that with further improvement, inverse AF could be used as universal approximation to any complicated posterior. Our analysis results in a unified approach to parameterizing a VAE, without the need to restrict ourselves to use factorial Gaussians in the latent real space.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “More discussion on this subject, and options for removing this restriction, are described in Huang et al. (2017) and Huang et al. (2018), and we defer studying more expressive elementwise densities to future work.”
Neural Autoregressive Flows (2018, background): “Our result builds on the work of Huang et al. (2017), who demonstrate the general universal representational capability of inverse autoregressive transformations parameterized by an autoregressive neural network (that transform uniform random variables into any random variables in reals).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “…as (1) an inference machine in the context of variational inference for continuous latent variable models (Kingma et al., 2016; Tomczak & Welling, 2016; Berg et al., 2018), and (2) a trainable component of the latent variable model (Chen et al., 2017; Agrawal & Dukkipati, 2016; Huang et al., 2017).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “transform distributions through a series of invertible parameterised functions [13], [61], [96], [124], [184], [189].”