NVAE: A Deep Hierarchical Variational Autoencoder

By Arash Vahdat, Jan Kautz. 2020

1,116 citations recorded by the source corpus.

Normalizing flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are among competing likelihood-based frameworks for deep generative learning. Among them, VAEs have the advantage of fast and tractable sampling and easy-to-access encoding networks. However, they are currently outperformed by other models such as normalizing flows and autoregressive models. While the majority of the research in VAEs is focused on the statistical challenges, we explore the orthogonal direction of carefully designing neural architectures for hierarchical VAEs. We propose Nouveau VAE (NVAE), a deep hierarchical VAE built for image generation using depth-wise separable convolutions and batch normalization. NVAE is equipped with a residual parameterization of Normal distributions and its training is stabilized by spectral regularization. We show that NVAE achieves state-of-the-art results among non-autoregressive likelihood-based models on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets and it provides a strong baseline on FFHQ. For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2.98 to 2.91 bits per dimension, and it produces high-quality images on CelebA HQ. To the best of our knowledge, NVAE is the first successful VAE applied to natural images as large as 256$\times$256 pixels. The source code is available at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Variational autoencoders (VAE) [45] and flow-based models [18, 19] enable efficient synthesis of high resolution images [9, 43, 89], but sample quality is not on par with GANs.”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Child, Vahdat and Kautz [5, 50] generate images using VAEs with a hierarchy of latent codes that increase progressively with resolution.”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “Concurrently to GANs, Variational Autoencoders (VAEs) [32, 57] have introduced a likelihood-based approach to image generation.”
Vector-quantized Image Modeling with Improved VQGAN (2021, background): “…Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2019b), Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Vahdat & Kautz, 2020), Diffusion Models (Song & Ermon, 2019; Dhariwal & Nichol, 2021) and Autoregressive Models (van den Oord et al., 2016; Parmar et…”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Our D2C implementation is based on the state-of-the-art NVAE [91] autoencoder structure, the U-Net diffusion model [42], and the MoCo-v2 contrastive representation learning method [16].”
Score-based Generative Modeling in Latent Space (2021, methodology): “The VAE’s encoder and decoder networks are trained using an Adamax optimizer [108], following NVAE [20].”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “VAEs are another promising class of likelihood-based models, and recent methods such as NVAE [63] and VDVAE [9] have successfully been applied to difﬁcult image generation domains.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…2014; Radford et al., 2015), variational autoencoders (VAEs) (Kingma & Welling, 2013; Kingma et al., 2016; Mittal et al., 2017; Marwah et al., 2017; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Ho et al., 2019a), Oord et al., 2017; Razavi et al.,…”
Image Super-Resolution via Iterative Refinement (2021, background): “Autoregressive models [5], [6], variational autoencoders (VAEs) [7], [8], Normalizing Flows (NFs) [9], [10], and GANs [11], [12], [13] have shown convincing image generation results and benefited conditional tasks such as image super-resolution [14], [15], [16], [17], [18].”
Taming Transformers for High-Resolution Image Synthesis (2020, neutral): “8 NVAE [59] 40.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “An alternative approach to stabilizing networks may be the spectral regularization method introduced in Vahdat & Kautz (2020).”
Dual Contradistinctive Generative Autoencoder (2020, neutral): “1 - NVAE† [61] Sampling - 50.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “In Table 6, we see that VAEBM has significantly higher AUROC than NVAE, justifying our argument that the energy function reduces the likelihood of non-data-like regions.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “NVAE [50] achieves the lowest LPIPS and MSE scores, though it requires orders of magnitude larger latent dimension compared to others.”
Improved Denoising Diffusion Probabilistic Models (2021, neutral): “(Ho et al., 2019) 3.69 3.08 PixelCNN (van den Oord et al., 2016c) 3.57 3.14 SPN (Menick & Kalchbrenner, 2018) 3.52 - NVAE (Vahdat & Kautz, 2020) - 2.91 Very Deep VAE (Child, 2020) 3.52 2.87 PixelSNAIL (Chen et al., 2018) 3.52 2.85 Image Transformer (Parmar et al., 2018) 3.48 2.90 Sparse Transformer ”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “In contrast, likelihood-based methods, such as Variational Autoencoders (VAEs) [29, 43], Diffusion Models [12, 24, 40] and Autoregressive Models [34, 44], offer distribution coverage and hence can generate more diverse samples [40,43, 44].”