Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS

By Yujia Xiao, Lei He, Huaiping Ming, Frank K. Soong. 2020

52 citations recorded by the source corpus.

Recent advances of neural TTS have made "human parity" synthesized speech possible when a large amount of studio-quality training data from a voice talent is available. However, with only limited, casual recordings from an ordinary speaker, human-like TTS is still a big challenge, in addition to other artifacts like incomplete sentences, repetition of words, etc. Chinese, a language, of which the text is different from that of other roman-letter based languages like English, has no blank space between adjacent words, hence word segmentation errors can cause serious semantic confusions and unnatural prosody. In this study, with a multi-speaker TTS to accommodate the insufficient training data of a target speaker, we investigate linguistic features and Bert-derived information to improve the prosody of our Mandarin Chinese TTS. Three factors are studied: phone-related and prosody-related linguistic features; better predicted breaks with a refined Bert-CRF model; augmented phoneme sequence with character embedding derived from a Bert model. Subjective tests on in- and out-domain tasks of News, Chat and Audiobook, have shown that all factors are effective for improving prosody of our Mandarin TTS. The model with additional character embeddings from Bert is the best one, which outperforms the baseline by 0.17 MOS gain.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Nima Mesgarani
7. Cong Han
8. Chenpeng Du

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “• Text pre-training [78, 101, 387, 140, 95, 447], which can provide better text representations by using pre-trained word embeddings or model parameters.”
• A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, background): “Xiao et al. [16] include Chinese BERT to improve prosody in Tacotron2.”
• PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “BERT, in particular, has proven effective in improving the performance of TTS models at either word level[4], character level [5, 6], or sentence level [7].”
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit (2020, methodology): “In [22, 23, 24], the textual knowledge from BERT [25] is introduced into TTS systems to improve the prosody of speech, but they ignore the variability of prosody.”
• Graphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis (2020, background): “Despite the progress [17, 18], Transformer TTS doesn’t explicitly associate input text with output utterances from syntactic point of view at sentence level, which is proven useful in speaking style and prosody modeling [19–22].”
• Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model (2020, methodology): “In [11, 13], large BERT models are used: 12-layers, and 768 hidden units in each layer (in [12] the size of the BERT models are not specified).”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, methodology): “Recent works [9, 16, 17, 19] have applied the BERT model as an auxiliary encoder for the TTS system.”
• Dependency Parsing based Semantic Representation Learning with Graph Neural Network for Enhancing Expressiveness of Text-to-Speech (2021, methodology): “Similar method further verifies the ability of BERT to improve prosody on the Chinese multi-speaker TTS task [15].”
• Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis (2020, methodology): “More recently, efforts have been made to use additional text embeddings derived from pre-trained LMs, such as the BERT model [25], to improve the modelling of prosody [26–28].”
• Acoustic Word Embeddings for End-to-End Speech Synthesis (2021, methodology): “[10,11] extract hidden states of BERT and pass them to the TTS model.”