Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

By Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, Nam Soo Kim. 2021

238 citations recorded by the source corpus.

Although neural text-to-speech (TTS) models have attracted a lot of attention and succeeded in generating human-like speech, there is still room for improvements to its naturalness and architectural efficiency. In this work, we propose a novel non-autoregressive TTS model, namely Diff-TTS, which achieves highly natural and efficient speech synthesis. Given the text, Diff-TTS exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via diffusion time steps. In order to learn the mel-spectrogram distribution conditioned on the text, we present a likelihood-based optimization method for TTS. Furthermore, to boost up the inference speed, we leverage the accelerated sampling method that allows Diff-TTS to generate raw waveforms much faster without significantly degrading perceptual quality. Through experiments, we verified that Diff-TTS generates 28 times faster than the real-time with a single NVIDIA 2080Ti GPU.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Sheng Zhao
7. Nima Mesgarani
8. Dong Yu

In Natural Language Processing

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Frank K. Soong
5. Dong Yu
6. Lei He
7. Yi Ren
8. Chao Weng

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Tao Qin
4. Zhou Zhao
5. Sheng Zhao
6. Jinglin Liu
7. Tie-Yan Liu
8. Rongjie Huang

In Signal Processing

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Tao Qin
5. Rongjie Huang
6. Tie-Yan Liu
7. Jinglin Liu
8. Nima Mesgarani

Citations in the corpus

• Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
• On Fast Sampling of Diffusion Probabilistic Models (2021, background): “, 2021; Lee & Han, 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), music synthesis (Liu et al.”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Some methods, utilizing techniques such as VAE (Hsu et al., 2019; Lee et al., 2022), flow (Miao et al., 2020; Kim et al., 2020), diffusion (Jeong et al., 2021; Kim et al., 2022; Popov et al., 2021), and others (Wu and Shi, 2022), have achieved promising performance in end-to-end speech synthesis.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Meanwhile, diffusion models have also been widely used in speech synthesis for waveform generation [28, 29] and acoustic feature generation [30, 31].”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “Remarkably, diffusion models [5, 6] have recently been shown to synthesize high-quality images in the image generation tasks and speech in the TTS synthesis tasks [7, 8, 9].”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “, 2021a; Nichol & Dhariwal, 2021), DDPM has also shown strong performance in speech synthesis (Chen et al., 2021a; Kong et al., 2021; Popov et al., 2021; Jeong et al., 2021).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Tacotron 1/2 [72, 73] Ph→MelS NAR / Hybrid/Self-Att/CNN MelNet [361] Ch→MelS AR / RNN DeepVoice [8] Ch/Ph→MelS AR / CNN DeepVoice 2 [84] Ch/Ph→MelS AR / CNN DeepVoice 3 [264] Ch/Ph→MelS AR Seq2Seq CNN ParaNet [262] Ph→MelS NAR Seq2Seq CNN DCTTS [326] Ch→MelS AR Seq2Seq CNN SpeedySpeech [355] Ph→MelS”
• Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “Most of the text-to-speech (TTS) methods [34], [35], [41] directly learn the mapping from text to wave samples or raw spectrogram pixels for the reason that the synthesized speech’s content relies on the words of text.”
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (2021, methodology): “There is a concurrent work (Jeong et al. 2021) at the submission time of our preprint which adopts a diffusion model as the acoustic model for TTS task.”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “Diffusion models have been demonstrated as a strong generative model for image generation [33], [34], [35], [36], text generation [37], [38] and audio generation [23], [39], [40], [41].”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, background): “Recently proposed Diff-TTS [15], Grad-TTS [31], and DiffSpeech [25] inherently are gradient-based models with score matching objectives to generate high-quality samples, while the iterative sampling process costs hinder their applications to text-to-speech deployment.”
• Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “, 2021), speech (Chen et al., 2021; Kong et al., 2021; Jeong et al., 2021), and music synthesis (Mittal et al.”
• Universal Speech Enhancement with Score-based Diffusion (2022, methodology): “In the audio domain, they have been particularly successful in speech synthesis [20, 21], text-to-speech [22, 23], bandwidth extension [24], or drum sound synthesis [25].”
• Deep Multimodal Data Fusion (2024, methodology): “Examples include the Diffusion-based text-to-speech model by [73] and various Diffusion-based image generation models such as DALL-E2 [144], Imagen [152], and Stable diffusion [151].”
• A Survey on Generative Diffusion Models (2022, methodology): “Guided-TTS [196] and Diff-TTS [193] employ components such as a speaker text encoder, duration predictor, and phoneme classifier for content generation and speech style guidance.”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Prior works [31, 85] generate Mel-spectrogram with a text-conditioned diffusion model.”
• Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, background): “Recent years have seen a surge in diffusion models as a leading approach for generating high-quality speech [2, 16, 27, 28, 10, 8].”