Preventing Posterior Collapse with delta-VAEs

By Ali Razavi, Aäron van den Oord, Ben Poole, Oriol Vinyals. 2019

191 citations recorded by the source corpus.

Due to the phenomenon of"posterior collapse,"current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data. In this paper, we propose an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information. Our proposed $\delta$-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate the efficacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet $32\times 32$.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “[80] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “It is known that expressive frame generative models are hard to condition [22, 23, 24, 25].”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, positive): “Our autoregressive priors are very similar to those described in (Razavi et al., 2019), which are in turn closely related to PixelSNAIL (Chen et al., 2018).”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, neutral): “…FFJORD [15] 3.40 - Flow++ [18] 3.08 3.86 Gated PixelCNN [35] 3.03 3.83 VFlow [4] 2.98 3.83 PixelCNN++ [40] 2.92 - NVAE [54] 2.91 3.92 Image Transformer [36] 2.90 3.77 Very Deep VAE [8] 2.87 3.80 PixelSNAIL [7] 2.85 3.80 δ -VAE [38] 2.83 3.77 Sparse Transformer [9] 2.80 - ScoreFlow (Ours) 2.83”
Variable-rate discrete representation learning (2021, methodology): “Finally, many recent works harnessing discrete representation learning for generative modelling (Dieleman et al., 2018; Razavi et al., 2019b; De Fauw et al., 2019; Dhariwal et al., 2020) use hierarchically structured representations as a strong inductive bias.”
Densely connected normalizing flows (2021, methodology): “Please note that we are unable to include approaches based on transformers [38, 39, 58] since they do not report the computational effort for model training.”
On Variational Learning of Controllable Representations for Text without Supervision (2019, background): “Several variations of VAEs have been proposed for better disentanglement (Higgins et al., 2017; Kumar et al., 2017; Chen et al., 2016; Razavi et al., 2019).”
Step-unrolled Denoising Autoencoders for Text Generation (2021, background): “This problem was later tackled by Delta-VAEs (Razavi et al., 2019), with limited success in the text domain (Bosc & Vincent, 2020).”