Discrete Flow Matching

By Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman. Neural Information Processing Systems, 2024

313 citations recorded by the source corpus.

Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions:(i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($\epsilon$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.

Subfields: Building and Construction, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a9ac6c4b165d8c4bf1a949d0931e7ca7a21fc55b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

1. Michihiro Yasunaga
2. Luke S. Zettlemoyer
3. Xuezhe Ma
4. Chunting Zhou
5. Jacob Kahn

Citations in the corpus

• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, background): “An interesting area of recent acrive research is the application diffusion models and their generalizations to discrete text generation [Li et al., 2022, Gat et al., 2024].”
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (2025, unknown): “Recent advancements in discrete diffusion models [2, 20, 21, 11, 3, 10, 18, 24, 31, 14, 42, 4, 37, 27, 28, 41, 5, 40, 39] have reshaped the landscape of text generation, offering a viable alternative to autoregressive (AR) paradigms in large language models (LLMs).”
• Large Language Diffusion Models (2025, unknown): “Lou et al. [17] showed that masked diffusion, as a special case of discrete diffusion, achieves perplexity comparable to or surpassing ARMs at GPT-2 scale.”