Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

By Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David C. Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann and 5 more. 2023

665 citations recorded by the source corpus.

While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Informatics

View Paper: https://www.semanticscholar.org/paper/985f0c89c5a607742ec43c1fdc2cbfe54541cbad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Health Informatics

Citations in the corpus

VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “We employ the MAGVIT-v2 [77] tokenizer for joint image and video tokenization, and the SoundStream [79] tokenizer for audio.”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “This can help achieve a higher compression ratio with largely improved quality and continuity of video reconstruction when compared to previous image VAEs (Rombach et al., 2022; Esser et al., 2021).”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “Inspired by the success and widespread adoption of binary vector quantization[80, 85], we scaled up the tokenizer vo-Index Quantized Feature cabulary to 2 64 , significantly surpassing all previous Au-toRegressive model vocabularies[58, 81].”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, positive): “To handle both videos and images, we adopt CausalConv3D [95].”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Different from [82], our latent representation is real-valued and quantization-free.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “In contrast, autoregressive (AR) models [17, 27, 38, 94, 104] generate videos sequentially, a paradigm that naturally aligns with the causal structure of temporal media.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “MagViT [73] adapts this approach to video data, and MagViT-2 [74] enhances MaskGIT by introducing an improved VQVAE.”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “However, such a simple quantization, unlike sophisticated tokenizers such as those designed for images [65, 72] and audio [19, 73], poses difficulties in action learning and limits action precision.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, neutral): “Orthogonal to this, RQ-VAE [36] and MoVQ [76] study the effect of using multiple vector quantization steps per latent embedding, while MAGVIT-v2 [68] and FSQ [45] propose a lookup-free quantization.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, neutral): “However, recent work [14, 44] identiﬁed that limitations in codebook utilization lead to an unintended trade-off: while larger code-books improve reconstruction ﬁdelity, they adversely affect generative performance.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “…alternative solutions for video compression and representation, including utilizing pre-trained checkpoints (e.g., compression network) [47], shortening the context window, using light-weight modeling mechanisms such as (grouped) multi-query attention [48, 49] or efficient…”