Vector Quantized Diffusion Model for Text-to-Image Synthesis

By Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo. 2021

1,090 citations recorded by the source corpus.

We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality. The code and models are available at https://github.com/cientgu/VQ-Diffusion.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, neutral): “Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “Concurrently with our work, Gu et al. (2021) train text-conditional discrete diffusion models on top of discrete latent codes, finding that the resulting system can produce competitive image samples.”
Scalable Diffusion Models with Transformers (2022, background): “They have also been trained on discrete codebooks [59] as both autoregressive models [11, 47] and masked generative models [4, 14]; the former has shown excellent scaling behavior up to 20B parameters [62].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “[21] used LAION-400M to train VQ diffusion text-to-image generation models, which have been shown to be more efficient, and are able to generate higher quality images.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “There are also opportunities to integrate scaled autoregressive models with diffusion models, starting with having an autoregressive model generate an initial low-resolution image and then iteratively refining and super-resolving images with diffusion modules [12, 13, 49].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, positive): “Inspire by [50, 51, 52], we first sample ˆX 0 from p θ ( X 0 | X t , X p , C ) , and then sample X t − ∆ t from q ( X t − ∆ t | ˆX 0 , X t ) , which involves remask ⌊ N · σ ( t − ∆ t ) ⌋ tokens in ˆX 0 with the lowest confidence score, where we define the confidence score of ˆ x i in ˆX 0 to p θ (ˆ…”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Our VQ-diffusion blocks, based on Transformer, largely follow the architecture in (Gu et al. 2022).”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, methodology): “6) Reparameterization Trick: Recent works [13], [47] found that approximating some surrogate variables, such as pθ(x̂0|xt,y) gives better results comparing with directly predicting the posterior q(xt−1|xt,x0).”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “The more details about discrete diffusion models, please refer to [23], [34], [45].”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, methodology): “[15] experiment with diffusion models in place of VQGAN.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, background): “When combined with pretrained text embeddings, text-to-image diffusion models [1, 42, 43, 44, 45, 46] are capable of generating high-fidelity images based on text prompts.”
Blended Latent Diffusion (2022, background): “Some recent works [Bond-Taylor et al. 2021; Esser et al. 2021b; Gu et al. 2021; Hu et al. 2021; Rombach et al. 2022; Vahdat et al. 2021] have thus proposed to perform the diffusion in a latent space with lower dimensions and higher-level semantics, compared to pixels, yielding competitive…”
UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models (2023, background): “Diffusion probabilistic models (DPMs) [33, 13, 35] have become the new prevailing generative models and have achieved competitive performance on many tasks including image synthesis [8, 29, 13], video synthesis [15], text-to-image generation [27, 29, 12], voice synthesis [5], etc .”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, neutral): “…et al., 2022a; Zhao et al., 2022), text-to-image generation (Nichol et al., 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Gu et al., 2022), voice synthesis (Liu et al., 2022a; Chen et al., 2021a;b), molecule generation (Xu et al., 2022; Hoogeboom et al., 2022; Wu et al.,…”
Text-to-3D using Gaussian Splatting (2023, background): “, 2023) or trains the diffusion model in latent space with an autoencoder (Rombach et al., 2022; Gu et al., 2022).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “VQ-Diffusion [78] proposes a vector-quantized diffusion model for text-to-image generation, and it eliminates the unidirectional bias and avoids accumulative prediction errors.”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “Inspired by Gu et al. (2022); Chang et al. (2022), we employ the discrete denoising diffusion to model discrete image tokens instead of continuous representations.”