DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training

By Hyung-Seok Oh, Sang-Hoon Lee, Seong-Whan Lee. 2023

35 citations recorded by the source corpus.

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior performance of our proposed method has been demonstrated via experiments.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Tao Qin
5. Tie-Yan Liu
6. Jinyu Li
7. Yichong Leng
8. Dongchao Yang

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “In specific, we select handcrafted prosody features (e.g., the first 20 bins of mel-spectrogram [7, 63, 64]) as the baseline.”
• Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters (2026, unknown): “…sentences and previously realized styles [7]; emotion-distribution predictors deduce multi-level emotion intensities solely from text [9]; and diffusion-based predictors replace deterministic regression with sampling, motivated by the one-to-many character of the text-to-prosody mapping [11], [12].”