Autoregressive Image Generation without Vector Quantization

By Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, Kaiming He. Neural Information Processing Systems, 2024

691 citations recorded by the source corpus.

Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while a discrete-valued space can facilitate representing a categorical distribution, it is not a necessity for autoregressive modeling. In this work, we propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space. Rather than using categorical cross-entropy loss, we define a Diffusion Loss function to model the per-token probability. This approach eliminates the need for discrete-valued tokenizers. We evaluate its effectiveness across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will motivate the use of autoregressive generation in other continuous-valued domains and applications. Code is available at: https://github.com/LTH14/mar.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/e46a79cf7c2c9192639f9e1d12280650e06b9d56

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “(2) Loss functions specifically designed for visual generation can be employed, such as diffusion loss [44].”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, positive): “As a result, some advanced autoregressive (AR) models [22, 39] are beginning to exhibit more competitive performance.”
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, background): “Thus, there have been studies exploring direct modeling in continuous space [27, 28, 29] to enhance synthesized speech quality recently.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “MAR[37], Fluid[23] proposed random-order models and employed continuous tokenizer rather than discrete tokenizer.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “We design a block-wise causal attention mechanism inspired by prior works that combine autoregressive models with diffusion [38, 40, 49, 110].”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Such design has been used in MAR-based [43] autoregressive video generation [111] and concurrently in other autoregressive video diffusion models [106, 107].”