Generating Diverse High-Fidelity Images with VQ-VAE-2

By Ali Razavi, Aäron van den Oord, Oriol Vinyals. 2019

2,413 citations recorded by the source corpus.

We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation. To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before. We use simple feed-forward encoder and decoder networks, making our model an attractive candidate for applications where the encoding and/or decoding speed is critical. Additionally, VQ-VAE requires sampling an autoregressive model only in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images. We demonstrate that a multi-scale hierarchical organization of VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that ha”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “[41] trains a multi-layer discrete autoencoder, allowing them to first sample coarse-grained latent codes and then use this as conditioning information when sampling higher-resolution latent codes.”
Jukebox: A Generative Model for Music (2020, methodology): “When using the hierarchical VQ-VAE from (Razavi et al., 2019) for raw audio, we observed that the bottlenecked top level is utilized very little and sometimes experiences a complete collapse, as the model decides to pass all information through the less bottlenecked lower levels.”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “We learn the model following a two-stage procedure similar to (van den Oord et al., 2017; Razavi et al., 2019).”
Zero-Shot Text-to-Image Generation (2021, methodology): “Oord et al. (2017); Razavi et al. (2019) address this using an online cluster assignment procedure coupled with the straight-through estimator (Bengio et al., 2013).”
Improved Techniques for Training Score-Based Generative Models (2020, background): “In particular, they can be sensitive to slight noise perturbations [23], as shown by the difference of”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Following (van den Oord et al., 2017; Habibian et al., 2019; Razavi et al., 2019) we model the latent distribution P( z ) using an ARM.”
Classifier-Free Diffusion Guidance (2022, methodology): “These models have delivered audio synthesis performance rivaling the quality of autoregressive models with substantially fewer inference steps [2, 9], and they have delivered ImageNet generation results outperforming BigGAN-deep [1] and VQ-VAE-2 [11] in terms of FID score and classification accuracy”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “VQ-VAE-2 [47] presented a hierarchical ar-3 chitecture composed of VQ-VAE models operating at multiple scales, enabling faster generation compared with pixel space generation.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “On the other hand, unconditional image models can synthesize photorealistic images (Brock et al., 2018; Karras et al., 2019a;b; Razavi et al., 2019), sometimes with enough ﬁdelity that humans can’t distinguish them from real images (Zhou et al., 2019).”
Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “The basic idea is to encode images into discrete latent tokens using VQ-VAE [39, 45], and pre-train a huge-size auto-regressive Transformers [46] to predict these discrete tokens based on paired text sequences.”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “C V ] 5 J un 2 02 2 Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al., 2017), DALLE (Ramesh et al., 2021) and VQGAN (Esser et al., 2021).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
Score-based Generative Modeling in Latent Space (2021, methodology): “In two-stage training, a separate generative model is trained in latent space as a new prior after training the VAE itself [60, 61, 62, 63, 64, 10].”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, methodology): “4 for a qualitative assessment of sample quality and diversity compared to VQ-VAE-2 [18] and BigGAN-deep [2].”
High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “in class-conditional [4, 24], text-conditional [23, 34] or pixelwise conditional settings [20, 33, 30].”