Improving Variational Auto-Encoders using Householder Flow

By J. Tomczak, Max Welling. 2016

187 citations recorded by the source corpus.

Variational auto-encoders (VAE) are scalable and powerful generative models. However, the choice of the variational posterior determines tractability and flexibility of the VAE. Commonly, latent variables are modeled using the normal distribution with a diagonal covariance matrix. This results in computational efficiency but typically it is not flexible enough to match the true posterior distribution. One fashion of enriching the variational posterior distribution is application of normalizing flows, i.e., a series of invertible transformations to latent variables with a simple posterior. In this paper, we follow this line of thinking and propose a volume-preserving flow that uses a series of Householder transformations. We show empirically on MNIST dataset and histopathology data that the proposed flow allows to obtain more flexible variational posterior and competitive results comparing to other normalizing flows.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Oncology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Oncology

In Generative Models

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Moreover, having access to tractable density is useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et al., 2017).”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “Here, the complementary research into learning a covariance structure through a framework of normalizing ﬂows [39, 52, 23] and the stacking of a hierarchy of latent variables [4, 37, 31, 50] have shown promising results.”
Neural Ordinary Differential Equations (2018, neutral): “Recent work explores the tradeoff between the expressiveness of normalizing ﬂ ow layers and computational cost (Kingma et al., 2016; Tomczak and Welling, 2016; Berg et al., 2018).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…the likelihood on a data sample, having access to tractable density is especially useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et…”
Neural Spline Flows (2019, methodology): “[20] also introduce linear transformations based on the QR-decomposition, where the orthogonal matrix is parameterized by a sequence of Householder transformations [53].”
VAE with a VampPrior (2017, background): “[20], Caltech 101 Silhouette [27], Frey Faces1 and Histopathology patches [39].”
Neural Autoregressive Flows (2018, background): “There Finally, there are also several works which develop new techniques for constructing NFs that are orthogonal to ours (Tomczak & Welling, 2017; 2016; Gemici et al., 2016; Du-venaud et al., 2016; Berg et al., 2018).”
Sylvester Normalizing Flows for Variational Inference (2018, neutral): “Householder Sylvester ﬂows can also be seen as a non-linear extension of Householder ﬂows [Tomczak and Welling, 2016].”
Taming VAEs (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [21, 22, 23, 24, 25, 3, 26, 5] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [27, 28].”
Inference Suboptimality in Variational Autoencoders (2018, background): “There has been significant work on improving inference in variational autoencoders (VAEs) [12, 19] through the development of expressive approximate posteriors [18, 11, 17, 23, 24].”
Analyzing Inverse Problems with Invertible Neural Networks (2018, methodology): “Besides variants of the original architectures [43, 2, 44], the most common realizations use auto-regressive flows, where the density is decomposed according to the Bayesian chain rule [23, 17, 12, 31, 29, 25, 38, 45].”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “Figure 3 shows the test log-likelihoods with various K on Histopathology.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “…have also been used as (1) an inference machine in the context of variational inference for continuous latent variable models (Kingma et al., 2016; Tomczak & Welling, 2016; Berg et al., 2018), and (2) a trainable component of the latent variable model (Chen et al., 2017; Agrawal & Dukkipati,…”
Generalized ELBO with Constrained Optimization , GECO (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [6, 7, 8, 9, 10, 11, 12, 13, 14] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [15, 16].”
Variational Item Response Theory: Fast, Accurate, and Expressive (2020, background): “There is a rich body of work in the machine learning community dedicated to richer posterior distributions for variational inference [44, 49, 4, 27, 30].”
Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference (2018, methodology): “Our setup is purposely identical to the setup of Tomczak & Welling (2016), where we use the dynamically binarized MNIST data set of Salakhutdinov & Murray (2008).”