Autoregressive Image Generation using Residual Quantization

By Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, Wook-Shin Han. 2022

851 citations recorded by the source corpus.

For autoregressive (AR) modeling of high-resolution images, vector quantization (VQ) represents an image as a sequence of discrete codes. A short sequence length is important for an AR model to reduce its computational costs to consider long-range interactions of codes. However, we postulate that previous VQ cannot shorten the code sequence and generate high-fidelity images together in terms of the rate-distortion trade-off. In this study, we propose the two-stage framework, which consists of Residual-Quantized VAE (RQ-VAE) and RQ-Transformer, to effectively generate high-resolution images. Given a fixed codebook size, RQ-VAE can precisely approximate a feature map of an image and represent the image as a stacked map of discrete codes. Then, RQ-Transformer learns to predict the quantized feature vector at the next position by predicting the next stack of codes. Thanks to the precise approximation of RQ-VAE, we can represent a $256\times 256$ image as $8\times 8$ resolution of the feature map, and RQ-Transformer can efficiently reduce the computational costs. Consequently, our framework out-performs the existing AR models on various benchmarks of unconditional and conditional image generation. Our approach also has a significantly faster sampling speed than previous AR models to generate high-quality images.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “…et al., 2022), ii) non-autoregressive, parallel decoding schemes (Gu et al., 2017; Ghazvininejad et al., 2019; Chang et al., 2022), iii) custom architectures adapted to the special structure of the tokens produced by neural audio codecs (Kreuk et al., 2022; Wang et al., 2023; Lee et al., 2022).”
Recommender Systems with Generative Retrieval (2023, methodology): “To the best of our knowledge, we are the first to use generative Semantic IDs created using an auto-encoder (RQ-VAE [20, 42]) for retrieval models.”
Building Normalizing Flows with Stochastic Interpolants (2022, neutral): “While progress has been from various angles with tools such as implicit maps (Goodfellow et al., 2014; Brock et al., 2019), and autoregressive maps (Menick & Kalchbrenner, 2019; Razavi et al., 2019; Lee et al., 2022), we focus on the case where the map has a clear associated probability ﬂow.”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, methodology): “The recursive quantization of RQ thus approximates the vector z in a coarse-to-fine manner [29].”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “…of models for text-to-image synthesis: GANbased (Tao et al., 2022; Zhu et al., 2019; Li et al., 2019; Fu et al., 2020; El-Nouby et al., 2019), autoregressive (Gu et al., 2022a; Lee et al., 2022; Ding et al., 2022) and diffusion models (Liu et al., 2021b; Nichol et al., 2021; Ruiz et al., 2022).”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “VQVAE-2 [52] and RQ-Transformer [37] also follow VQGAN’s raster-scan autoregressive method but further improve VQVAE via extra scales or stacked codes.”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “, 2019, 2020), and an auto-regressive model (RQ-VT; Lee et al., 2022).”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, neutral): “VQ-based methods [21, 35, 50, 58, 67] employ vector quantization to convert image patches into index-wise tokens and use a decoder-only transformer to predict the next token index.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, neutral): “Orthogonal to this, RQ-VAE [36] and MoVQ [76] study the effect of using multiple vector quantization steps per latent embedding, while MAGVIT-v2 [68] and FSQ [45] propose a lookup-free quantization.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “VILA-U utilizes residual quantization [29] to improve the representational capacity of discrete visual features - incorporating a depth transformer, as introduced in RQ-VAE [29], to gradually predict the residual tokens.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “…of autoregressive models on large language models, pioneering works attempt to explore autoregressive models in image generation, for example, VQVAE [Van Den Oord et al. 2017; Razavi et al. 2019], VQGAN [Esser et al. 2021; Lee et al. 2022], DALL-E [Ramesh et al. 2021], Parti [Yu et al. 2021, 2022].”
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “Language models have also demonstrated their ability to model high-level, long-term sequences for different content types, as shown by the recent advances in text [40, 41, 42, 43, 44] and image [45, 46, 47, 48, 49, 50].”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, methodology): “This can be attributed to VQ being difﬁcult to optimize, whereas FSQ can be viewed as the standard VQ formulation changed such that a) VQ Alternatives Residual quantization (RVQ) has been used for image (Lee et al., 2022) and audio (Zeghidour et al., 2021) generation.”
Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, methodology): “In this section, we present the proposed LLM-based recommendation model LC-Rec , which integrates L anguage and C ollaborative semantics for improving LLMs in Rec ommender systems.”
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024, methodology): “In reinforcement learning, Lee et al. (2022) use cubic masking patches to corrupt video sequences as an auxiliary task for learning a state representation model.”