Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis

By Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuanbin Cao, Heiga Zen, Yonghui Wu. 2020

131 citations recorded by the source corpus.

This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer level representations on coarser level ones. Additionally, it imposes hierarchical conditioning across all latent dimensions using a conditional variational auto-encoder (VAE) with an auto-regressive structure. Evaluation of reconstruction performance illustrates that the new structure does not degrade the model while allowing better interpretability. Interpretations of prosody attributes are provided together with the comparison between word-level and phone-level prosody representations. Moreover, both qualitative and quantitative evaluations are used to demonstrate the improvement in the disentanglement of the latent dimensions.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “For example, [11, 12, 13] uses phoneme-level prosodies, [14] uses word-level prosodies and [15, 16] uses hierarchical prosodies.”
A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “Two different VAE models are investigated; a global VAE per speaker similar to [28] and a fine-grained phoneme-level VAE akin to [30].”
Controllable Neural Prosody Synthesis (2020, background): “Recently, significant effort has gone into disentangling the latent spaces of high-quality neural vocoders to recover explicit prosodic control [16, 17, 18, 19].”
Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis (2021, background): “The work of [26] targets interpretable and controllable hierarchical prosodic controls and comes closest to the approach we pursue.”
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Sun et al. [42, 41] adopt VAE to represent the ﬁne-grained prosody variable, naturally enabling sampling of different prosody features for each phoneme.”
Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, background): “While some of the approaches [13, 14, 15, 16, 17] rely on prosodic features such as fundamental frequency, duration and energy to generate the expressive speech but these approaches can generate expressive speech for the speakers who are already part of the training.”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “However, most of them focused not on cloning a target speaker but on controlling the disentangled attributes [15, 16, 17, 18, 19, 22, 23], and some of them could not synthesize speech of an unseen speaker due to their architectural limitations [21].”
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “The first class is to provide additional acoustic or linguistic conditions for the acoustic model except the phoneme sequence input, such as pitch, GST embedding or phoneme embeddings [60, 61, 62], prosodies in different granularities have also been added to TTS in [63, 64].”
Mixture Density Network for Phone-Level Prosody Modelling in Speech Synthesis (2021, methodology): “Similar prior works [10, 11, 12] model the distribution of e with a single Gaussian in VAE.”
Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis (2021, background): “[21] proposes hierarchical VAE for phone-level prosody modelling, which tries to obtain an interpretable latent space.”
Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, methodology): “We instead utilize a ﬁne-grained VAE (FVAE) similar to Sun et al. (2020); Lee & Kim (2019) to model the alignment between the input tokens and the target mel-spectrogram frames, and extract per-token latent features from the target mel-spectrogram based on this alignment.”
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech (2021, unknown): “In order to resolve such a problem, several approaches [7, 8, 9, 10, 11] has been explored upon TTS models so that they can synthesize and control speech with various style.”
Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit (2020, methodology): “In [20], an interpretable latent variable model for prosody based on Tacotron 2 is presented to model phoneme-level and word-level prosody information of speech.”
Rich Prosody Diversity Modelling with Phone-Level Mixture Density Network (2021, background): “Hierarchical and quantized versions of VAE for phone-level prosody modelling is also investigated in [11, 12, 13, 14], which improves the interpretability and naturalness in synthetic speech.”
Expressive TTS Training With Frame and Style Reconstruction Loss (2020, background): “[34], [35] study a way to include a hierarchical, fine-grained prosody representation, that represents the recent attempts to”