Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

By Chunting Zhou, Lili Yu, A. Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke S. Zettlemoyer, Omer Levy. arXiv.org, 2024

446 citations recorded by the source corpus.

We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens. By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches. We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.

Subfields: Neurology

View Paper: https://www.semanticscholar.org/paper/5b3991fe7d8f6fc0a7fbd42938e2988ea37efe14

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “Unified integration of LLM and diffusion models within a single transformer [40, 50, 66, 104].”
Hallucination of Multimodal Large Language Models: A Survey (2024, unknown): “This integration of human feedback into the training loop has demonstrated effectiveness in enhancing the alignment of LLMs.”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “The results of other methods in the GenEval are sourced from [26, 92, 108, 22].”
π0: A Vision-Language-Action Flow Model for General Robot Control (2024, unknown): “Our architecture is inspired by Transfusion [59], which trains a single transformer using multiple objectives, with tokens corresponding to continuous outputs supervised via a flow matching loss and tokens corresponding to discrete outputs supervised via a cross-entropy loss.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, unknown): “For multimodal understanding, we use the SigLIP [89] encoder to extract high-dimensional semantic features from images.”
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “Very recently, hybrid models integrating autoregressive and diffusion frameworks have emerged as a promising direction in generative modeling of videos [8, 16, 20, 22, 28, 33, 45, 50, 52, 89, 100, 106, 107] as well as other sequence domains [1, 12, 43, 53, 59, 90, 110].”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, unknown): “Recent advances in unified multimodal models [55, 59, 63, 73] suggest promising directions for improvements.”
OmniGen: Unified Image Generation (2024, unknown): “Concurrent works such as TransFusion [79] and Show-O [71] unify diffusion and autoregressive methods into a single model, generating text autoregressively and images through diffusion.”