PixelVAE: A Latent Variable Model for Natural Images

By Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, Aaron C. Courville. 2016

362 citations recorded by the source corpus.

Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.

Subfields: Cancer Research, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Neural Discrete Representation Learning (2017, methodology): “PixelCNNs [29, 38] are convolutional autoregressive models which have also been used as distribution in the decoder of VAEs [15, 7].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “This has spawned a new class of explicit density models that adds an autoregressive component to the generative process of a latent variable model [14, 5].”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “And there is also a line of work that combined Helmholtz Machines with autoregressive models for images (Chen et al., 2016; Gulrajani et al., 2016) and for text data (Chung et al.”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Attempts to ﬁx this properties by including explicit latent variables [24, 7, 19] have not been overly successful, mainly due the expressiveness of AR models, providing little incentive for learning additional latent variables.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “It is possible to have autoregressive p θ ( x | z ) (Gulrajani et al., 2016), but generation is slow for these models.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “Depending on the use case, the encoder can process the whole utterance, emit a single latent vector and feed it to an autoregressive decoder [33], [46] or the encoder can periodically emit vectors of latent features to be consumed by the decoder [19], [47].”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Unfortunately, VAEs with powerful decoders suffer from posterior collapse [6, 9, 18, 46]: they will often neglect to use the latent variables at all, because the regularisation term in the loss function explicitly discourages this.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “Autoregressive models have also been combined with VAEs as decoder models (Gulrajani et al., 2016).”
Variational Lossy Autoencoder (2016, neutral): “Concurrent with our work, PixelVAE (Gulrajani et al., 2016) also explored using conditional Pixel-CNN as a VAE’s decoder and has obtained impressive density modeling results through the use of multiple levels of stochastic units.”
Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders (2017, methodology): “The joint probablity is now: We could parameterize Z as a latent variable p ( Z | x ) that we would have to marginalize over (Gulrajani et al., 2016), but in practice we have found this to be less eﬀective.”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “ion learning, Chen et al. [19] also explore the orthogonal direction of using a learned prior based on autoregressive ﬂow [56] to improve generative modeling capabilities of VLAE. PixelVAE. PixelVAEs [14] use a VAE with feed-forward convolutional encoder and decoder, combining the decoder with a (sh”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, methodology): “…(e.g. by reducing its spatial resolution, using a latent vector without spatial structure, inserting a VQ bottleneck and/or introducing architectural constraints in the encoder) or limiting the receptive ﬁeld of the decoder (Gulrajani et al., 2016; van den Oord et al., 2017; Dieleman et al., 2018).”
VAE with a VampPrior (2017, background): “53 on dynamic MNIST, and the hierarchical PixelVAE [15] that gets −78.”
Structured Disentangled Representations (2018, background): “ility distribution p (xjz) parameterized by a deep neural network. There have been many recent successes in training deep generative models for complex data types such as images [Gatys et al., 2015, Gulrajani et al., 2017], audio [Oord et al., 2016], and language [Bowman et al., 2016]. The latent co”
Taming VAEs (2018, background): “Deep variational auto-encoders (VAEs) have made substantial progress in the last few years on modelling complex data such as images [1, 2, 3, 4, 5, 6, 7], speech synthesis [8, 9], molecular discovery [10, 11, 12], robotics [13, 14, 15] and 3d-scene understanding [16, 17, 18].”
Preventing Posterior Collapse with delta-VAEs (2019, background): “Most common approaches either change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017), or weaken the decoder (Bowman et al., 2015; Gulrajani et al., 2016).”