BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

By Lars Maaløe, Marco Fraccaro, Valentin Liévin, Ole Winther. 2019

228 citations recorded by the source corpus.

With the introduction of the variational autoencoder (VAE), probabilistic latent variable models have received renewed attention as powerful generative models. However, their performance in terms of test likelihood and quality of generated samples has been surpassed by autoregressive models without stochastic units. Furthermore, flow-based models have recently been shown to be an attractive alternative that scales well to high-dimensional data. In this paper we close the performance gap by constructing VAE models that can effectively utilize a deep hierarchy of stochastic variables and model complex covariance structures. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path. We show that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution. We observe that BIVA, in contrast to recent results, can be used for anomaly detection. We attribute this to the hierarchy of latent variables which is able to extract high-level semantic features. Finally, we extend BIVA to semi-supervised classification tasks and show that it performs comparably to state-of-the-art results by generative adversarial networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble ﬂows [9, 43, 10, 30, 5, 14, 21] and VAEs [31, 44, 34], diffusion models are designed so that q has no parameters and the top-level latent x T has nearly zero mutual information with the data x 0 .”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Unfortunately, even the deepest instantiations of these models [44, 68, 8] lack in sample quality compared to other generative models and are oftentimes restricted to highly regular datasets.”
Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “Maaløe et al. (2019), for instance, incorporates a “bottom-up” stochastic path that doubles the depth of the approximate posterior, and Vahdat & Kautz (2020) introduces a number of powerful architecture components and improved training techniques.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, methodology): “B c B c Batch Normalization: The state-of-the-art VAE [4, 36] models have omitted BN as they observed that “the noise introduced by batch normalization hurts performance” [4] and have relied on weight normalization (WN) [46] instead.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “Currently, hierarchical VAEs attain theoretical code lengths comparable to NFs (Maaløe et al., 2019; Ho et al., 2019a) and DPMs achieve the best code lengths (Kingma et al.”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, background): “The framework of variational autoencoders [18, 24] (VAEs) and its extensions (e.g. [3, 17, 26, 20]) offers a scalable set of techniques for learning deep latent-variable models and corresponding inference models.”
Bilateral Denoising Diffusion Models (2021, neutral): “On the other hand, for those using the surrogate losses, such as the evidence lower bound in variational auto-encoders [17–19] and the contrastive divergence in energy-based models [20, 21], they improve inference speed but often yield lower quality samples than autoregressive and GAN models and…”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “…generalizes VAEs and hierarchical VAEs, one potential direction is to consider parameterizations that have shared components which are shown to be conducive to training, such as the ResNet with top-downn inference (Kingma et al., 2016) and the bidirectional inference machine (Maaløe et al., 2019).”
Automated curricula through setter-solver interactions (2019, background): “Adding the challenge of trying to discover latent variables with which to condition and performing even a simple manipulation of them makes things much more difﬁcult (Rezende & Viola, 2018) (c.f. the difﬁculty of learning hierarchies of latent variables (Sønderby et al., 2016; Maaløe et al., 2019)).”
Further Analysis of Outlier Detection with Deep Generative Models (2020, unknown): “Building on these concerns, various diagnostics have been deployed to evaluate the calibration of newly proposed DGMs [5–9], or applied when revisiting older modeling practices [10].”
Improved Conditional VRNNs for Video Prediction (2019, methodology): “We mitigate this problem by using a warmup regime for the KL loss [33] and a dense connectivity pattern [14, 24] between the input and latent variables.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, unknown): “LAE (Pidhorskyi et al., 2020) 19.21 PGGAN (Karras et al., 2017) 8.03 NVAE-Recon (Vahdat & Kautz, 2020) (Vahdat & Kautz, 2020) 78.67 BIVA (Maaløe et al., 2019) 78.41 DAVE++ (Vahdat et al., 2018b) 78.49 IAF-VAE (Kingma et al., 2016) 79.10 VampPrior AR dec. (Tomczak & Welling) 78.45 model is far from m”
Compression with Flows via Local Bits-Back Coding (2019, background): “VAEs and flow models (RealNVP-type models specifically) currently attain similar theoretical codelengths on image datasets [19, 29] and have similarly fast coding algorithms, but VAEs are more difficult to train due to posterior collapse [4], which implies worse net codelengths unless carefully tune”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “Other approaches have explored changing the VAE network architecture to help alleviate posterior collapse; for example adding skip connections [30, 15]”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, unknown): “Future work could investigate sharing parameters with the generative model as in Maaløe et. al’s Bidirection inference VAE (BIVA) [24].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “By introducing skip connections around the stochastic sampling process, latents can be conditioned on all previously sampled latents [125], [147], [227].”
IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression (2020, background): “Examples include autoregressive neural networks [36, 42], variational autoencoders [26, 41, 45, 31] and normalizing flows [47, 46, 10, 41, 11, 37, 27, 21, 15, 14, 20, 17], with the latter two enjoying faster decoding times.”