Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images

By Rewon Child. 2020

404 citations recorded by the source corpus.

We present a hierarchical VAE that, for the first time, outperforms the PixelCNN in log-likelihood on all natural image benchmarks. We begin by observing that VAEs can actually implement autoregressive models, and other, more efficient generative models, if made sufficiently deep. Despite this, autoregressive models have traditionally outperformed VAEs. We test if insufficient depth explains the performance gap by by scaling a VAE to greater stochastic depth than previously explored and evaluating it on CIFAR-10, ImageNet, and FFHQ. We find that, in comparison to the PixelCNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied to high-resolution images. We visualize the generative process and show the VAEs learn efficient hierarchical visual representations. We release our source code and models at https://github.com/openai/vdvae.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Variational autoencoders (VAE) [45] and flow-based models [18, 19] enable efficient synthesis of high resolution images [9, 43, 89], but sample quality is not on par with GANs.”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “2, we explain why these approaches do not necessarily remove large “prior holes”, so their sample qualities remain relatively poor even after many layers [91, 19].”
Score-based Generative Modeling in Latent Space (2021, methodology): “The VAE’s encoder and decoder networks are trained using an Adamax optimizer [108], following NVAE [20].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2015), variational autoencoders (VAEs) (Kingma & Welling, 2013; Kingma et al., 2016; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al.”
Improved Denoising Diffusion Probabilistic Models (2021, neutral): “…3.69 3.08 PixelCNN (van den Oord et al., 2016c) 3.57 3.14 SPN (Menick & Kalchbrenner, 2018) 3.52 - NVAE (Vahdat & Kautz, 2020) - 2.91 Very Deep VAE (Child, 2020) 3.52 2.87 PixelSNAIL (Chen et al., 2018) 3.52 2.85 Image Transformer (Parmar et al., 2018) 3.48 2.90 Sparse Transformer (Child et al.,…”
GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, methodology): “We use nearest-neighbor upsampling over transposed convolution as [24, 25] report various advantages (e.”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, neutral): “…FFJORD [15] 3.40 - Flow++ [18] 3.08 3.86 Gated PixelCNN [35] 3.03 3.83 VFlow [4] 2.98 3.83 PixelCNN++ [40] 2.92 - NVAE [54] 2.91 3.92 Image Transformer [36] 2.90 3.77 Very Deep VAE [8] 2.87 3.80 PixelSNAIL [7] 2.85 3.80 δ -VAE [38] 2.83 3.77 Sparse Transformer [9] 2.80 - ScoreFlow (Ours) 2.83”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “DDPMs are also very closely related to variational autoencoders [Kingma and Welling, 2013], where more recent work has shown that, with many stochastic layers, they can also attain competitive negative log likelihoods in unconditional image generation [Child, 2020].”
Perception Prioritized Training of Diffusion Models (2022, neutral): “…FID ↓ FFHQ BigGAN ICLR’19 [3] GAN 12.4 UNet GAN CVPR’20 [37] GAN 10.9 StyleGAN2 CVPR’20 [21] GAN 3.73 NVAE NeurIPS’20 [44] VAE 26.02 VDVAE ICLR’21 [5] VAE 33.5 VQGAN CVPR’21 [11] GAN+AR 9.6 D2C NeurIPS’21 [38] Diff [20] GAN 64.70 MSG-GAN CVPR’20 [16] [22] Flow 68.93 ALAE CVPR’20 [33] GAN 19.21…”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “The models we evaluate represent popular classes of generative models, including a variational autoencoder (VDVAE; Child, 2020), a flow model (Dense-”