Variational Lossy Autoencoder

By Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel. 2016

713 citations recorded by the source corpus.

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Neural Discrete Representation Learning (2017, methodology): “As the work in [7] suggests, the best generative models (as measured by log-likelihood) will be those without latents but a powerful decoder (such as PixelCNN).”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “Although inverse-autoregressive flows (IAFs) and autoregressive models can in principle model the same distributions (Chen et al., 2016), they have different inductive biases and may vary greatly in their capacity to model certain processes.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “Fitting prior distributions using neural networks from training data has become common practice, as it can significantly improve the performance of latent variable models [6].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “This has spawned a new class of explicit density models that adds an autoregressive component to the generative process of a latent variable model [14, 5].”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “And there is also a line of work that combined Helmholtz Machines with autoregressive models for images (Chen et al., 2016; Gulrajani et al., 2016) and for text data (Chung et al.”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Attempts to ﬁx this properties by including explicit latent variables [24, 7, 19] have not been overly successful, mainly due the expressiveness of AR models, providing little incentive for learning additional latent variables.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Since the evidence lower bound (ELBO) for VAEs do not necessarily promote informative latent representations [15], D2C comes with the benefit of being able to learn more informative latents despite not exactly optimizing an ELBO.”
Score-based Generative Modeling in Latent Space (2021, background): “For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as energy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “They also sometimes ignore latent variables entirely, becoming equivalent to normal autoregressive models (Chen et al. (2016)).”
Diagnosing and Enhancing VAE Models (2019, methodology): “…henceforth denoted as 2-Stage VAE , against three baseline VAE models diﬀering only in the decoder output layer: a Gaussian layer with ﬁxed γ , a Gaussian layer with a learned γ , and a cross-entropy layer as has been adopted in several previous applications involving images (Chen et al., 2016b).”
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, background): “Normalizing ﬂows (Rezende & Mohamed, 2015; Kingma et al., 2016) make the posterior more complex by composing a simple Gaussian posterior with an invertible smooth mapping for which the determinant of the Jacobian is tractable.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Unfortunately, VAEs with powerful decoders suffer from posterior collapse [6, 9, 18, 46]: they will often neglect to use the latent variables at all, because the regularisation term in the loss function explicitly discourages this.”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, background): “2 Empirically, VAEs with highly flexible priors or highly flexible approximate posteriors can obtain values of L that are near their own log-likelihood (Kingma et al., 2016; Chen et al., 2016b).”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “model is expressive, a vanilla VAE sacrifices log-likelihood in favor of amortized inference [12].”
Scalable Recollections for Continual Lifelong Learning (2017, background): “This setting poses a signiﬁcant challenge for the scalable recollection module since the CIFAR tiny image domain is known to be a particularly difﬁcult setting for VAE performance (Kingma et al., 2016; Chen et al., 2017).”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “Rezende & Mohamed (2015), Chen et al. (2017) and Ziegler & Rush (2019) improve VAE performance by enhancing the expressive power of prior and posterior distribution with normalizing ﬂows.”