Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit

By Zhen Zeng, Jianzong Wang, Ning Cheng, Jing Xiao. 2020

9 citations recorded by the source corpus.

Recent neural speech synthesis systems have gradually focused on the control of prosody to improve the quality of synthesized speech, but they rarely consider the variability of prosody and the correlation between prosody and semantics together. In this paper, a prosody learning mechanism is proposed to model the prosody of speech based on TTS system, where the prosody information of speech is extracted from the melspectrum by a prosody learner and combined with the phoneme sequence to reconstruct the mel-spectrum. Meanwhile, the sematic features of text from the pre-trained language model is introduced to improve the prosody prediction results. In addition, a novel self-attention structure, named as local attention, is proposed to lift this restriction of input text length, where the relative position information of the sequence is modeled by the relative position matrices so that the position encodings is no longer needed. Experiments on English and Mandarin show that speech with more satisfactory prosody has obtained in our model. Especially in Mandarin synthesis, our proposed model outperforms baseline model with a MOS gap of 0.08, and the overall naturalness of the synthesized speech has been significantly improved.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Frank K. Soong
6. Yanqing Liu
7. Bohan Li
8. Lei He

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Information Granularity Language/Speaker Level Multi-lingual/speaker TTS [438, 241, 38] Paragraph Level Long-form reading [11, 389, 370] Utterance Level Timbre/Prosody/Noise [303, 377, 139, 315, 202, 39] Word/Syllable Level Fine-grained information [319, 113, 44, 329] Character/Phoneme Level [318, 4”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “…of 2 convolutional layers with filter size and kernel size of 256 and 3 respectively, and a linear layer to compress the hidden to a dimension of 4 (we choose the dimension of 4 according to our preliminary study and is also consistent with previous works (Sun et al., 2020; Zeng et al., 2020)).”
• The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “Zeng et al. (2020) used relative attention in speech synthesis, letting each query interact with separate matrix transformations for each key vector, depending on their relative-distance offset.”