FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

By Rongjie Huang, Max W. Y. Lam, J. Wang, Dan Su, Dong Yu, Yi Ren, Zhou Zhao. 2022

237 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthesis. This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. A noise schedule predictor is also adopted to reduce the sampling steps without sacrificing the generation quality. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms without any intermediate feature (e.g., Mel-spectrogram). Our evaluation of FastDiff demonstrates the state-of-the-art results with higher-quality (MOS 4.28) speech samples. Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time. We further show that FastDiff generalized well to the mel-spectrogram inversion of unseen speakers, and FastDiff-TTS outperformed other competing methods in end-to-end text-to-speech synthesis. Audio samples are available at https://FastDiff.github.io/.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rongjie Huang
2. Zhou Zhao
3. Nima Mesgarani
4. Yossi Adi
5. Wei-Ning Hsu
6. Jinglin Liu
7. Yi Ren
8. Shinji Watanabe

In Generative Models

1. Zhou Zhao
2. Yi Ren
3. Rongjie Huang
4. Jinglin Liu
5. Yossi Adi
6. Wei-Ning Hsu
7. Avi Schwarzschild
8. Micah Goldblum

In Speech & Audio

1. Yossi Adi
2. Rongjie Huang
3. Wei-Ning Hsu
4. Jinglin Liu
5. Zhou Zhao
6. Yi Ren
7. Shinji Watanabe
8. Nima Mesgarani

In Signal Processing

1. Rongjie Huang
2. Yossi Adi
3. Zhou Zhao
4. Nima Mesgarani
5. Jinglin Liu
6. Yi Ren
7. Wei-Ning Hsu
8. Shinji Watanabe

Citations in the corpus

• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “Despite the success of large-scale generative models in other areas, most speech models are still trained on datasets at the scale of tens to hundreds of hours [51, 31, 32, 46, 24, 58, 7].”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Despite being diffusion-based, our model is faster than VITS, FastDiff [28], and ProDiff [18], two of the fastest diffusion-based TTS models, with 5 iterations of diffusion and iSTFTNet decoder.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Adapting models (Chen et al., 2020b; Huang et al., 2022) to a specific individual or object is a long-standing goal in machine learning research.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, background): “Denoising diffusion probabilistic models (DDPMs) [8, 23, 44] are likelihood-based generative models that have recently succeeded to advance the state-of-the-art results in several important domains including image synthesis [5, 44], audio synthesis [11], and 3D point cloud generation [27], and have ”
• Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, background): “Among them, FastDiff has achieved remarkable results in high-quality speech synthesis [8].”
• M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, unknown): “Recently, benefited from the rapid development of deep learning and the continuous breakthroughs in TTS techniques [47; 23; 52; 37; 35; 36; 18], singing voice synthesis [45; 21; 14; 5; 38; 2; 27; 40; 9; 13; 10; 51; 17; 8; 43; 1; 22; 48; 54; 41; 19; 56; 53; 25] and its related tasks (e.g., SVC [33;…”
• Universal Guidance for Diffusion Models (2023, background): “The approach has been successfully extended to a number of domains, such as audio and text generation [1, 11, 13, 14].”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Pioneering work WaveGrad [7] Code / Project DiffWave [45] Code Efficient vocoder BDDM [48] Code InferGrad [9] WaveFit [43] Project Statistical improvement DDGM [70] PriorGrad [50] Project ItôWave [125] Project SpecGrad [44] End-to-end Pioneering work WaveGrad 2 [8] Code / Project CRASH [90] Project ”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, background): “As illustrated in Figure 1, similar to text-to-speech (TTS) systems [4, 8, 9, 27, 30], SVS systems [1, 6, 18, 28] generally adopt an acoustic model to convert the musical score 1 into the intermediate features (i.”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “There has been an increasing interest in self-supervised learning in the machine learning (Zhang et al., 2022a; Lam et al., 2021; Zhang et al., 2023b; 2022b) and multimodal processing community (Xia et al., 2022; Zhang et al., 2023a; Zhang et al.; Huang et al., 2022b;a).”
• DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation (2023, background): “The recently explored Diffusion Probabilistic Models (DPMs) [4] serve as a powerful TTS generative backbone, achieving surprising results [5, 6, 7].”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Recently, non-autoregressive TTS models [23, 12, 18, 17] have been proposed and significantly speed up mel-spectrogram generation, and researchers have experimented on style modeling and transferring in parallel: Meta-StyleSpeech [31] generally adopts a speech encoding network for multi-speaker TTS ”
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “Despite the successful applications in text processing and generation, replicating this success for audio modality (speech (Ren et al., 2020; Huang et al., 2022a), music (Huang et al., 2021; Liu et al., 2022a), sound (Yang et al., 2022; Huang et al., 2023a), and talking head (Wu et al., 2021; Ye et…”