Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
By Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang. Neural Information Processing Systems, 2024
1,081 citations recorded by the source corpus.
We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine"next-scale prediction"or"next-resolution prediction", diverging from the standard raster-scan"next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes GPT-like AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/76e8218f657c77c38da44daaed5bb54ab727a8fc
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, positive): “Possible unified modeling solutions for pyramids include hierarchical architectures (Rombach et al., 2022; Crowson et al., 2024; Hatamizadeh et al., 2024) or via next-token prediction (Yu et al., 2023; Tian et al., 2024), but involve architectural changes.”
- BLOCK DIFFUSION: INTERPOLATING BETWEEN AU-TOREGRESSIVE AND DIFFUSION LANGUAGE MODELS (2025, neutral): “…iterative editing (Gu et al., 2019), parallel and speculative decoding (Gu et al., 2017; Santilli et al., 2023; Cai et al., 2024; Gloeckle et al., 2024), consistency training (Kou et al., 2024), guidance (Sanchez et al., 2023), and cross-modal extensions (Liu et al., 2023; Tian et al., 2025).”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “In the case of hindsight control, the entire control sequence is provided as input, with some actions masked, and the model is trained to recover these masked actions.”
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, neutral): “As a result, some advanced autoregressive (AR) models [22, 39] are beginning to exhibit more competitive performance.”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “More recently, some works have explored autoregressive methods for vision genera-† tion [70, 76], achieving performance comparable to diffusion models.”
- Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “For example, MaskGIT [Chang et al. 2022], MAGVIT [Yu et al. 2023a,b] adopt the masked image modeling strategy, VAR [Tian et al. 2024] uses hierarchical multi-scale property.”
- Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “However, directly enlarging the vocabulary in existing tokenizers[58, 64, 71, 81] leads to a substantial increase in memory consumption and computational burden.”