Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis

By Yukiya Hono, Kazuna Tsuboi, Kei Sawada, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, Keiichi Tokuda. 2020

27 citations recorded by the source corpus.

This paper proposes a hierarchical generative model with a multi-grained latent variable to synthesize expressive speech. In recent years, fine-grained latent variables are introduced into the text-to-speech synthesis that enable the fine control of the prosody and speaking styles of synthesized speech. However, the naturalness of speech degrades when these latent variables are obtained by sampling from the standard Gaussian prior. To solve this problem, we propose a novel framework for modeling the fine-grained latent variables, considering the dependence on an input text, a hierarchical linguistic structure, and a temporal structure of latent variables. This framework consists of a multi-grained variational autoencoder, a conditional prior, and a multi-level auto-regressive latent converter to obtain the different time-resolution latent variables and sample the finer-level latent variables from the coarser-level ones by taking into account the input text. Experimental results indicate an appropriate method of sampling fine-grained latent variables without the reference signal at the synthesis stage. Our proposed framework also provides the controllability of speaking style in an entire utterance.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei-Ning Hsu
2. Yossi Adi
3. Xu Tan
4. Tao Qin
5. Tie-Yan Liu
6. Emmanuel Dupoux
7. Jade Copet
8. Abdel-rahman Mohamed

In Signal Processing

1. Wei-Ning Hsu
2. Yossi Adi
3. Xu Tan
4. Jade Copet
5. Emmanuel Dupoux
6. Tao Qin
7. Tu Nguyen
8. Abdel-rahman Mohamed

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Information Granularity Language/Speaker Level Multi-lingual/speaker TTS [438, 241, 38] Paragraph Level Long-form reading [11, 389, 370] Utterance Level Timbre/Prosody/Noise [303, 377, 139, 315, 202, 39] Word/Syllable Level Fine-grained information [319, 113, 44, 329] Character/Phoneme Level [318, 4”
• Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis (2021, background): “Prosody has a top-down hierarchical structure [8], [9], [10], including discourse, phrase, utterance, word, syllable and phone.”
• Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “…speech synthesis models have shown great success in recent years (Skerry-Ryan et al., 2018a; Ren et al., 2020; Kim et al., 2021; Sorin et al., 2020; Hono et al., 2020), the majority is still text-based, i.e., Text-to-Speech (Tan et al., 2021), and is therefore limited in terms of expressivity.”
• Mixture Density Network for Phone-Level Prosody Modelling in Speech Synthesis (2021, background): “Hierarchical and quantized versions of VAE for PL prosody modelling is also investigated in [11, 12, 13], which improves the interpretability and naturalness in synthetic speech.”