Self-conditioned Embedding Diffusion for Text Generation

By Robin Strudel, Corentin Tallec, Florent Altch'e, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre and 1 more. 2022

134 citations recorded by the source corpus.

Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in a continuous space of embeddings, as is standard in language modeling. We propose Self-conditioned Embedding Diffusion, a continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models - while being in theory more efficient on accelerator hardware at inference time. Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements to continuous diffusion.

Subfields: Artificial Intelligence, Generative Models, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Natural Language Generation

Ting-Li Chen

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, methodology): “, 2021) and has been adopted in several existing work (Strudel et al., 2022; Dieleman et al., 2022; Chen et al., 2022a).”
A Survey on Generative Diffusion Models (2022, background): “Itincorporatesenhancedloss functions[172],[176],[177],diversegenerationtypes[175], [178],andadvancedmodelarchitectures[15],[173].”
Continuous diffusion for categorical data (2022, methodology): “Diffusion models on the other hand require a fixed-size ‘canvas’ which is iteratively refined during sampling, although inserting padding tokens at random during training (and collapsing them after sampling) could allow for some length variation (Strudel et al., 2022), and it”
Large Language Diffusion Models (2025, unknown): “A simple approach is to continuousize text data and apply continuous diffusion models directly [41– 51].”
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, unknown): “For generation, we assess sample quality using perplexity (PPL) on unconditional samples measured by an additional larger language model (i.e., GPT-2 large), and sample diversity through unigram entropy (Strudel et al., 2022).”
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, unknown): “…2022) and classifier-free guidance (Ho, 2022) can further improve generation quality, although recent work has shown that methods like self-conditioning (Strudel et al., 2022) and designing a less sparse embedding space (e.g. based on bits) (Chen et al., 2022) can obviate the need for such methods.”