Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis
By Guanghui Xu, Wei Song, Zhengchen Zhang, Chao Zhang, Xiaodong He, Bowen Zhou. 2020
56 citations recorded by the source corpus.
Although speech prosody is related to the linguistic information up to the discourse structure, most text-to-speech (TTS) systems only take into account the information within each sentence. This makes it challenging when converting a paragraph of text into natural and expressive speech. In this paper, we propose to use the text embeddings of the neighboring sentences to improve the prosody generation for each utterance of a paragraph in an end-to-end fashion without using any explicit prosody features. More specifically, cross-utterance (CU) context vectors, which are produced by an additional CU encoder based on the sentence embeddings extracted by a pretrained BERT model, are used to augment the input of the Tacotron2 decoder. Two types of BERT embeddings are investigated, which leads to the use of different CU encoder structures. Experimental results on a Mandarin audiobook dataset and the LJ-Speech English audiobook dataset demonstrate the use of CU information can improve the naturalness and expressiveness of the synthesized speech. Subjective listening testing shows most of the participants prefer the voice generated using the CU encoder over that generated using standard Tacotron2. It is also found that the prosody can be controlled indirectly by changing the neighbouring sentences.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- A Survey on Neural Speech Synthesis (2021, background): “Information Granularity Language/Speaker Level Multi-lingual/speaker TTS [438, 241, 38] Paragraph Level Long-form reading [11, 389, 370] Utterance Level Timbre/Prosody/Noise [303, 377, 139, 315, 202, 39] Word/Syllable Level Fine-grained information [319, 113, 44, 329] Character/Phoneme Level [318, 4”
- PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS (2021, methodology): “Such pre-training has been applied to TTS for improving the performance both in low-resource scenarios [19] and high-resource scenarios [20–22], by using embeddings at subword-level [20, 21], word-level [19], or sentence-level [21, 22].”
- A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “[17] propose using BERT embeddings to generate context vectors from neighboring sentences to improve prosody modelling.”
- DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, methodology): “DigitalObjectIdentifier 10.1109/TASLP.2024.3395994 In some studies [8], [9], reference encoders have been used to extract prosody vectors for prosody modeling.”
- PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “BERT, in particular, has proven effective in improving the performance of TTS models at either word level[4], character level [5, 6], or sentence level [7].”
- SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, background): “…the recent development in TTS that the text and audio contexts from the surrounding utterances are useful to achieve more natural prosody modelling and enable the use of more natural and casual speech data (Xu et al., 2021; Guo et al., 2021; Oplustil-Gallegos et al., 2021; Zhang et al., 2023c).”
- Improving Speech Prosody of Audiobook Text-To-Speech Synthesis with Acoustic and Textual Contexts (2022, methodology): “Note that, different from most previous works [9, 10, 11], we set the context length of each lateral to k characters instead of setting it to a certain number of sentences, by which we can easily evaluate model performance with different context lengths.”