Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

By Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, Zehuan Yuan. arXiv.org, 2024

767 citations recorded by the source corpus.

We introduce LlamaGen, a new family of image generation models that apply original ``next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly. We reexamine design spaces of image tokenizers, scalability properties of image generation models, and their training data quality. The outcome of this exploration consists of: (1) An image tokenizer with downsample ratio of 16, reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet benchmark. (2) A series of class-conditional image generation models ranging from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256 benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A text-conditional image generation model with 775M parameters, from two-stage training on LAION-COCO and high aesthetics quality images, demonstrating competitive performance of visual quality and text alignment. (4) We verify the effectiveness of LLM serving frameworks in optimizing the inference speed of image generation models and achieve 326% - 414% speedup. We release all models and codes to facilitate open-source community of visual generation and multimodal foundation models.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/b15e6e2b1d81bc110f8fc98c3caf2e25e2512539

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “…(Rombach et al., 2022), GigaGAN (Kang et al., 2023), PixArt (Chen et al., 2024), Imagen (Saharia et al., 2022), RAPHAEL Xue et al. (2024), LlamaGen (Sun et al., 2024), SDXL (Podell et al., 2023), and SD3 (Esser et al., 2024); unified models including Gemini (Anil et al., 2023), NExT-GPT (Wu et…”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, unknown): “…Flow [24] 17.20 + torch.compile & bf16 [38] - 100k × 1024 16.59 + AdamW smaller beta2 [1] 16.61 + Logit Normal Sampling [8] 13.99 + Velocity Direction Loss [43] 12.52 + Gradient Clip (1.0) [5] 14.40 + SwiGLU FFN [35] 10.10 + RMS Norm [46] 9.25 + Rotary Pos Embed [36] 7.13 + f16 & patchsize=1 5.58”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, unknown): “Inspired by the success and widespread adoption of binary vector quantization[80, 85], we scaled up the tokenizer vo-Index Quantized Feature cabulary to 2 64 , significantly surpassing all previous Au-toRegressive model vocabularies[58, 81].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, unknown): “For visual generation tasks, we use the VQ tokenizer from [70] to convert images into discrete IDs.”