Vector-quantized Image Modeling with Improved VQGAN

By Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, Yonghui Wu. 2021

809 citations recorded by the source corpus.

Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vector-quantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at \(256\times256\) resolution, we achieve Inception Score (IS) of 175.1 and Fr'echet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. VIM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “It is learned with the same losses of ViT-VQGAN (perceptual loss, StyleGAN loss and `2 loss with same loss weighting in [21]), mapping from reconstructed images to higher-resolution reconstructed images.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “[79] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu.”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “A contemporary work to ours, VIM [51], proposes to use a VIT backbone [13] to further improve the tokenization stage.”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “This technique ensures that each code embedding contains less information, facilitating information disentanglement [32, 46].”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “The L2-normalization of the encoded and codebook vectors converts euclidean distance to cosine similarity, which is helpful for stability and quality [43].”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “This approach has been used to generate high-resolution images [11], [12], [42] and long videos [43].”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “(xiii) image-token prediction objective, whereby we tokenize WebLI images (256×256 resolution) using a ViT-VQGAN [29] model with patch size 16×16 (256 tokens per image); this objective is framed as a 2D masked-token task (i.e., fill-in the missing grid pieces, with the corresponding image pixels als”
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, methodology): “V I T-VQGAN [54] improved class-conditioned image synthesis with codebook improvements and parameterizing VQGAN with ViT [9].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “In addition to the VQ objective, we adopt the factorized and (cid:96) 2 - normalized codes from ViT-VQGAN [58] to improve codebook usage and reconstruction quality.”
Recommender Systems with Generative Retrieval (2023, positive): “Notably, similar ideas have been adopted when generating images from text (e.g., Parti [41]), where images are represented as tokens using ViT-VQGAN [40].”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “A simple trick for training a larger codebook involves decreasing the code embedding dimen-sion when increasing the vocabulary size (Yu et al., 2022a).”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, background): “Later GAN variants have focused on progressive generation (Zhang et al., 2017; Hong et al., 2018), or better text-image alignment (Xu et al., 2018; Zhang et al., 2021).”
Muse: Text-To-Image Generation via Masked Generative Transformers (2023, methodology): “ViT-VQGAN (Yu et al., 2021) extends VQGAN to the Transformer architecture and introduces techniques like factorized codes and l 2 normalized codes to improve codebook usage.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “…of autoregressive models on large language models, pioneering works attempt to explore autoregressive models in image generation, for example, VQVAE [Van Den Oord et al. 2017; Razavi et al. 2019], VQGAN [Esser et al. 2021; Lee et al. 2022], DALL-E [Ramesh et al. 2021], Parti [Yu et al. 2021, 2022].”
Visual Prompting via Image Inpainting (2022, methodology): “To tackle the multimodal nature of synthesis, different approaches have been proposed to model the distribution over possible completions [15, 57, 7].”
Grounding Language Models to Images for Multimodal Generation (2023, methodology): “, 2022) through model scaling, pretraining, and improved image quantization models (Esser et al., 2021; Yu et al., 2021).”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, background): “…as a successful paradigm for image generation, outperforming GANs (Xu et al., 2017; Zhu et al., 2019; Zhang et al., 2021) and comparable to autoregressive models (Ramesh et al., 2021; Yu et al., 2021; Esser et al., 2020; Yu et al., 2022; Ding et al., 2021) in terms of ﬁdelity and diversity.”