ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech

By Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, Yi Ren. 2022

259 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hinder their applications to text-to-speech deployment. Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling. In this work, we propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. Unlike previous work estimating the gradient for data density, ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. To tackle the model convergence challenge with decreased diffusion iterations, ProDiff reduces the data variance in the target site via knowledge distillation. Specifically, the denoising model uses the generated mel-spectrogram from an N-step DDIM teacher as the training target and distills the behavior into a new model with N/2 steps. As such, it allows the TTS model to make sharp predictions and further reduces the sampling time by orders of magnitude. Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms, while it maintains sample quality and diversity competitive with state-of-the-art models using hundreds of steps. ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU, making diffusion models practically applicable to text-to-speech synthesis deployment for the first time. Our extensive ablation studies demonstrate that each design in ProDiff is effective, and we further show that ProDiff can be easily extended to the multi-speaker setting.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nima Mesgarani
2. Rongjie Huang
3. Zhou Zhao
4. Jinglin Liu
5. Cong Han
6. Yi Ren
7. Chenye Cui
8. Zhuo Chen

In Generative Models

1. Zhou Zhao
2. Yi Ren
3. Rongjie Huang
4. Jinglin Liu
5. Bin Cui
6. Yue Zhao
7. Zhuo Chen
8. Qiang Yang

In Speech & Audio

1. Jinglin Liu
2. Rongjie Huang
3. Zhou Zhao
4. Yi Ren
5. Chenye Cui
6. Nima Mesgarani
7. Zhuo Chen
8. Cong Han

In Signal Processing

1. Nima Mesgarani
2. Rongjie Huang
3. Zhou Zhao
4. Jinglin Liu
5. Yi Ren
6. Cong Han
7. Chenye Cui
8. Zhuo Chen

Citations in the corpus

• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “By leveraging diffusion models and their variants (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song and Ermon, 2020; Song et al., 2021; Lipman et al., 2023), many works have successfully applied them to the audio domain (Popov et al., 2021; Huang et al., 2022, 2023; Shen et al., 2023).”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Despite being diffusion-based, our model is faster than VITS, FastDiff [28], and ProDiff [18], two of the fastest diffusion-based TTS models, with 5 iterations of diffusion and iSTFTNet decoder.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “In this study, we focus on using audio codec models to help solve generation problems in audio-related works, such as Text-to-Speech (TTS) [1–4], music generation [5], audio generation [6–10].”
• DiffusionDet: Diffusion Model for Object Detection (2022, background): “…models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications [2,40,44,78,94,99,102], etc.”
• DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation (2023, background): “The recently explored Diffusion Probabilistic Models (DPMs) [4] serve as a powerful TTS generative backbone, achieving surprising results [5, 6, 7].”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “According to ProDiff [28], estimating the gradient of data density (gradient-based parameterization) is dominant in previous diffusion models, which requires hundreds or thousands of iterations for high quality generation [28].”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Text-to-speech [39, 37, 20, 13] aims to generate almost human-like audios using text, which attracts broad interest in the machine learning community.”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “…works bring NAR model into other sequence-to-sequence tasks (Cui et al., 2021; Ye et al., 2023; Huang et al., 2022c; Yang et al., 2022b), such as video caption (Yang et al., 2019), speech recognition (Chen et al., 2020) and speech synthesis (Ye et al., 2022; Huang et al., 2022d; Yang et al., 2023).”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Instead of estimating the gradient of data density, ProDiff [99] parameterizes the denoising diffusion model by directly predicting the clean data.”