Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

By Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov. 2021

759 citations recorded by the source corpus.

Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.

Subfields: Statistical and Nonlinear Physics, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

1. Arash Vahdat
2. Karsten Kreis
3. Yang Song
4. Jun Zhu
5. Stefano Ermon
6. Chongxuan Li
7. William Chan
8. Xu Tan

In Generative Models

1. Arash Vahdat
2. Karsten Kreis
3. Yang Song
4. Stefano Ermon
5. Jun Zhu
6. Jonathan Ho
7. Chongxuan Li
8. Jan Kautz

In Speech & Audio

1. Xu Tan
2. Jinyu Li
3. Tao Qin
4. Yi Ren
5. Zhou Zhao
6. Sheng Zhao
7. Tie-Yan Liu
8. Lei He

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Dong Yu
4. Tao Qin
5. Yi Ren
6. Nima Mesgarani
7. Zhou Zhao
8. Tie-Yan Liu

Citations in the corpus

• On Fast Sampling of Diffusion Probabilistic Models (2021, background): “, 2021; Lee & Han, 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), music synthesis (Liu et al.”
• Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “With these important improvements, DPMs show great potential in various applications, including speech synthesis (Chen et al., 2020; Kong et al., 2020; Popov et al., 2021; Lam et al., 2021), controllable generation (Choi et al.”
• Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “They are also rapidly finding use in other domains such as audio [27, 37] and video [19] generation, image segmentation [4, 54] and language translation [34].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Previous works using raw waveform [8, 9] and mel-spectrogram [1, 2, 10, 7, 11] as data representations suffer from these intricate complexities during speech generation.”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In the garden, where the flowers that the gardener who retired last year still bloomed, the children who play there every afternoon find peace and joy.”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Some methods, utilizing techniques such as VAE (Hsu et al., 2019; Lee et al., 2022), flow (Miao et al., 2020; Kim et al., 2020), diffusion (Jeong et al., 2021; Kim et al., 2022; Popov et al., 2021), and others (Wu and Shi, 2022), have achieved promising performance in end-to-end speech synthesis.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Despite the success of large-scale generative models in other areas, most speech models are still trained on datasets at the scale of tens to hundreds of hours [51, 31, 32, 46, 24, 58, 7].”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Additional techniques, such as normalizing flow (Valle et al. 2020; Kim et al. 2020) and diffusion models (Liu et al. 2022; Popov et al. 2021; Liu, Guo, and Yu 2023), have been introduced to generate the mel-spectrogram.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
• PolyVoice: Language Models for Speech to Speech Translation (2023, background): “In recent years, neural text-to-speech (TTS) synthesis has achieved significant developments, and the progress of neural network structure makes continuous improvements in the intelligence of synthetic speech (Wang et al., 2017; Ren et al., 2019; Kim et al., 2021; Popov et al., 2021).”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “During inference, for the diffusion model, we find it beneficial to use a temperature τ and sample the terminal condition zT from N (0, τ−1I) [31].”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Owing to the development of non-autoregressive generative models, recent TTS models can generate high-quality speech with faster inference speed (Ren et al., 2019; 2021; Kim et al., 2020; Popov et al., 2021).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Tacotron 1/2 [72, 73] Ph→MelS NAR / Hybrid/Self-Att/CNN MelNet [361] Ch→MelS AR / RNN DeepVoice [8] Ch/Ph→MelS AR / CNN DeepVoice 2 [84] Ch/Ph→MelS AR / CNN DeepVoice 3 [264] Ch/Ph→MelS AR Seq2Seq CNN ParaNet [262] Ph→MelS NAR Seq2Seq CNN DCTTS [326] Ch→MelS AR Seq2Seq CNN SpeedySpeech [355] Ph→MelS”
• Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “Diffusion generative models were first proposed in [11] and achieved strong results on image generation [13], [31], [32], [33] and speech synthesis [34], [35], [36], [37].”
• Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “Score-based generative models [44, 45, 48] and diffusion probabilistic models [43, 19] have recently achieved state-of-the-art sample quality in a number of tasks, including image generation [48, 11], audio synthesis [5, 27, 37], and shape generation [3].”
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (2021, background): “There is also a concurrent work [24] which adjusts diffusion model as the acoustic model for TTS task.”