Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis

By Chung-Ming Chien, Hung-yi Lee. 2020

41 citations recorded by the source corpus.

Prosody modeling is an essential component in modern text-to-speech (TTS) frameworks. By explicitly providing prosody features to the TTS model, the style of synthesized utterances can thus be controlled. However, predicting natural and reasonable prosody at inference time is challenging. In this work, we analyzed the behavior of non-autoregressive TTS models under different prosody-modeling settings and proposed a hierarchical architecture, in which the prediction of phoneme-level prosody features are conditioned on the word-level prosody features. The proposed method outperforms other competitors in terms of audio quality and prosody naturalness on objective and subjective evaluation.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Frank K. Soong
6. Yi Ren
7. Zhou Zhao
8. Lei He

In Experimental and Cognitive Psychology

1. Tao Qin
2. Chenpeng Du
3. Frank K. Soong
4. Xie Chen
5. Tie-Yan Liu
6. Yiwei Guo
7. Sheng Zhao
8. K. Yu

Citations in the corpus

• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “For example, [11, 12, 13] uses phoneme-level prosodies, [14] uses word-level prosodies and [15, 16] uses hierarchical prosodies.”
• A Survey on Neural Speech Synthesis (2021, background): “5) Character/phoneme level [318, 423, 319, 44, 39, 185], such as duration, pitch or prosody information.”
• Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, methodology): “To generate expressive speech, reference-encoder-based methods [10, 11, 12, 13] use autoencoder-like architecture to learn latent disentangled representations and successfully factorize speaker identity and prosody; prosody prediction-based methods [5, 14] first extract prosody attributes including…”
• Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis (2021, background): “Prosody has a top-down hierarchical structure [8], [9], [10], including discourse, phrase, utterance, word, syllable and phone.”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “The first class is to provide additional acoustic or linguistic conditions for the acoustic model except the phoneme sequence input, such as pitch, GST embedding or phoneme embeddings [60, 61, 62], prosodies in different granularities have also been added to TTS in [63, 64].”
• Rich Prosody Diversity Modelling with Phone-Level Mixture Density Network (2021, background): “Hierarchical and quantized versions of VAE for phone-level prosody modelling is also investigated in [11, 12, 13, 14], which improves the interpretability and naturalness in synthetic speech.”