DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models

By Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, Lingpeng Kong. 2022

592 citations recorded by the source corpus.

Recently, diffusion models have emerged as a new paradigm for generative models. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is under-explored due to the discrete nature of texts, especially for conditional generation. We tackle this challenge by proposing DiffuSeq: a diffusion model designed for sequence-to-sequence (Seq2Seq) text generation tasks. Upon extensive evaluation over a wide range of Seq2Seq tasks, we find DiffuSeq achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models. Apart from quality, an intriguing property of DiffuSeq is its high diversity during generation, which is desired in many Seq2Seq tasks. We further include a theoretical analysis revealing the connection between DiffuSeq and autoregressive/non-autoregressive models. Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks. Code is available at \url{https://github.com/Shark-NLP/DiffuSeq}

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Dong Yu
3. Jinyu Li
4. Tie-Yan Liu
5. Tao Qin
6. Jianfeng Gao
7. Sheng Zhao
8. Li Deng

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Xu Tan
4. Rongjie Huang
5. Sheng Zhao
6. Chao Weng
7. Dongchao Yang
8. Stefano Ermon

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Since X 0 is unavailable in inference, we use the diffusion model p θ , parameterized by θ , to predict the masked tokens conditioned on X p and C , denoted as p θ ( X 0 | X t , X p , C ) .”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Diffusion models have been demonstrated as a strong generative model for image generation [33]–[36], text generation [37], [38] and audio generation [23], [39]–[41].”
• DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “Another line of work studies diffusion models for discrete data by applying continuous diffusion models with Gaussian noise on the embedding space of discrete data [30, 69, 24], the {− 1 .”
• Dream 7B: Diffusion Large Language Models (2025, unknown): “Building on their remarkable success, a number of works have extended diffusion techniques to text generation tasks by modeling text in a continuous embedding space, building continuous diffusion language models (Li et al., 2022; Gong et al., 2023b,a; Dieleman et al., 2022).”
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (2023, background): “…prior works improve downstream performance with alternative loss functions (moving away from likelihood-based score matching) (Han et al., 2022; Mahabadi et al., 2023) and explicitly encoding conditional information (e.g. inputting an infilling mask) (Gong et al., 2023; Dieleman et al., 2022).”
• DiffusionDet: Diffusion Model for Object Detection (2022, background): “…models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications [2,40,44,78,94,99,102], etc.”
• Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “…diffusion models have demonstrated superior performance across a range of generation tasks in various modalities, including text-to-image generation [40, 43, 41, 17], image manipulation [26, 34, 4, 56], video synthesis [18, 47, 22], text generation [28, 16, 59], 3D avatar synthesis [39, 53], etc.”
• Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “In this section, we introduce RAG foundations and outline enhancement methods that further improve the effectiveness.”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “DiffuSeq [75] conditions on texts and proposes a seq-to-seq diffusion framework that helps with four NLP tasks.”
• DeepCache: Accelerating Diffusion Models for Free (2023, background): “These models have demonstrated remarkable efficacy across diverse applications, being employed for the generation of images [21, 62, 66], text [11, 30], audio [6, 46], and video [19, 38, 58].”
• Large Language Diffusion Models (2025, unknown): “A simple approach is to continuousize text data and apply continuous diffusion models directly [41– 51].”
• A Survey on Generative Diffusion Models (2022, unknown): “Itincorporatesenhancedloss functions[172],[176],[177],diversegenerationtypes[175], [178],andadvancedmodelarchitectures[15],[173].”