Avoiding Latent Variable Collapse With Generative Skip Models

By Adji Bousso Dieng, Yoon Kim, Alexander M. Rush, David M. Blei. 2018

187 citations recorded by the source corpus.

Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," especially if the likelihood model is powerful. Specifically, the lower bound involves an approximate posterior of the latent variables; this posterior "collapses" when it is set equal to the prior, i.e., when the approximate posterior is independent of the data. While VAEs learn good generative models, latent variable collapse prevents them from learning useful representations. In this paper, we propose a simple new way to avoid latent variable collapse by including skip connections in our generative model; these connections enforce strong links between the latent variables and the likelihood function. We study generative skip models both theoretically and empirically. Theoretically, we prove that skip models increase the mutual information between the observations and the inferred latent variables. Empirically, we study images (MNIST and Omniglot) and text (Yahoo). Compared to existing VAE architectures, we show that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “A related idea was recently proposed by [7], that add skip connections among the neural network layers parameterizing a shallow VAE with a single latent variable.”
Preventing Posterior Collapse with delta-VAEs (2019, background): “(2016), and the posterior collapse problems by several others (e.g. van den Oord et al., 2017; Kim et al., 2018; Dieng et al., 2018).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “(Yang et al., 2017) ≤ 332.1 10.0 – – ≤ 359.1 7.6 – – SA-VAE + anneal (Kim et al., 2018) ≤ 327.5 7.19 – – – – – – Modiﬁed VAE Objective VAE + anneal 328.6 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 357.9 (0.1) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) β -VAE ( β = 0.2) 332.2 (0.6) 19.1 (1.5) 3.3 (0.1) 20.4 (6.8) 360.7…”
Prescribed Generative Adversarial Networks (2019, methodology): “Maximizing VAE ( θ , φ ) is equivalent to minimizing this KL which leads to issues such as latent variable collapse (Bowman et al., 2015; Dieng et al., 2018b).”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, background): “Skip connections were developed in (Dieng et al., 2018) to shorten the paths from z to x in the decoder.”
Mixture Content Selection for Diverse Sequence Generation (2019, unknown): “Dieng et al., 2018; Xu and Durrett, 2018; He et al., 2019; Razavi et al., 2019).”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “Other approaches have explored changing the VAE network architecture to help alleviate posterior collapse; for example adding skip connections [30, 15]”
Consistency Regularization for Variational Auto-Encoders (2021, background): “Researchers have looked at ways to improve the quality of the latent representations of vaes, often tackling the so-called latent variable collapse problem—in which the approximate posterior distribution induced by the encoder collapses to the prior over the latent variables (Bowman et al., 2015; Ki”
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, background): “, 2019), decoder architectures (Yang et al., 2017; Dieng et al., 2018), auxiliary loss (Zhao et al.”
AAVAE: Augmentation-Augmented Variational Autoencoders (2021, background): “The ﬁrst perspective is the so-called posterior collapse [34, 16], in which all the approximate posterior distributions, i.e., the representation vectors, are detached from the input and collapse onto each other.”
Practical and Consistent Estimation of f-Divergences (2019, background): “Several recent papers have estimated or optimized this quantity in the context of autoencoder architectures, coinciding with our setting [8, 19, 1, 31].”