Diffsound: Discrete Diffusion Model for Text-to-Sound Generation

By Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, Dong Yu. 2022

434 citations recorded by the source corpus.

Generating sound effects that people want is an important topic. However, there are limited studies in this area for sound generation. In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a token-decoder, and a vocoder. The framework first uses the token-decoder to transfer the text features extracted from the text encoder to a mel-spectrogram with the help of VQ-VAE, and then the vocoder is used to transform the generated mel-spectrogram into a waveform. We found that the token-decoder significantly influences the generation performance. Thus, we focus on designing a good token-decoder in this study. We begin with the traditional autoregressive (AR) token-decoder. However, the AR token-decoder always predicts the mel-spectrogram tokens one by one in order, which may introduce the unidirectional bias and accumulation of errors problems. Moreover, with the AR token-decoder, the sound generation time increases linearly with the sound duration. To overcome the shortcomings introduced by AR token-decoders, we propose a non-autoregressive token-decoder based on the discrete diffusion model, named Diffsound. Specifically, the Diffsound model predicts all of the mel-spectrogram tokens in one step and then refines the predicted tokens in the next step, so the best-predicted results can be obtained by iteration. Our experiments show that our proposed Diffsound model not only produces better generation results when compared with the AR token-decoder but also has a faster generation speed, i.e., MOS: 3.56 v.s 2.786.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “We use Librilight [56], which contains 60 K hours of 16 KHz unlabeled speech data and around 7000 distinct speakers from LibriVox audiobooks, as the training set.”
Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “They are capable of creating and infilling complex acoustic scenes like “birds chirping and water dripping with some banging in the background” given a text description (Yang et al., 2023c; Kreuk et al., 2022; Huang et al., 2023b; Ghosal et al., 2023; Liu et al., 2023b,c).”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…prompt speech Singing Voice Synthesis (SVS) (Liu et al., 2022) phoneme (with duration), speaker prompt, MIDI singing Text-to-Sound (TT-Sound) (Yang et al., 2023c) textual description sounds Text-to-Music (TT-Music) (Agostinelli et al., 2023) textual description music Audio Edit (A-Edit) ♣♢…”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, background): “Similarly, many works [4, 12] try to use VQ-VAE model to compress the time-frequency spectrogram (e.”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Academic and industrial communities also pay increasing attention to synthesizing speech or sound from text, namely text-to-speech (TTS) [Li et al., 2019, Ren et al., 2019] or text-to-sound [Yang et al., 2022, Kreuk et al., 2022].”
AudioGen: Textually Guided Audio Generation (2022, methodology): “Compared to the existing text-to-audio work (Yang et al., 2022), A UDIO G EN generates samples that obtain better objective and subjective metrics.”
MusicLM: Generating Music From Text (2023, methodology): “In contrast, and inspired by progress in text-to-image generation (Ramesh et al., 2021; 2022; Saharia et al., 2022; Yu et al., 2022), recent work has explored generating audio from sequence-wide, high-level captions (Yang et al., 2022; Kreuk et al., 2022) such as “whistling with wind blowing” .”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “The more details about discrete diffusion models, please refer to [23], [34], [45].”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “Following common practice (Yang et al., 2022; Iashin & Rahtu, 2021), the key automated performance metrics used are melception-based (Koutini et al., 2021) FID (Heusel et al., 2017) and KL divergence to measure audio ﬁdelity.”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “…models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications [2,40,44,78,94,99,102], etc.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Diffsound [261] presents a non-autoregressive decoder based on the discrete diffusion model [6, 215], which predicts all the mel-spectrogram tokens in every single step, and then refines the predicted tokens in the following steps.”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, positive): “…discrete state spaces [1, 14, 15] and have been successfully applied to applications ranging from graph generation [16], text-to-sound generation [17] or protein design [18], they remain not as widely used as their continuous counterparts as they are not competitive with autoregressive models in…”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “as audio LLMs [76, 77, 78] and vision LLMs [78, 79, 80].”
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “To benchmark our models, we also included two state-of-the-art audio generation models, namely DiffSound [18] and AudioGen [16].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “, 2022a), sound (Yang et al., 2022; Huang et al., 2023a), and talking head (Wu et al.”
DiGress: Discrete Denoising diffusion for graph generation (2022, background): “Several works have recently proposed discrete diffusion models for text, images, audio, and attributed point clouds (Austin et al., 2021; Yang et al., 2022; Luo et al., 2022).”