Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model

By Tom Kenter, Manish Sharma, R. Clark. 2020

42 citations recorded by the source corpus.

The prosody of currently available speech synthesis systems can be unnatural due to the systems only having access to the text, possibly enriched by linguistic information such as part-of-speech tags and parse trees. We show that incorporating a BERT model in an RNN-based speech synthesis model — where the BERT model is pretrained on large amounts of unlabeled data, and fine-tuned to the speech domain — improves prosody. Additionally, we propose a way of handling arbitrarily long sequences with BERT. Our findings indicate that small BERT models work better than big ones, and that fine-tuning the BERT part of the model is pivotal for getting good results.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Ye Jia
6. Chenpeng Du
7. Yu Zhang
8. Nima Mesgarani

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “…that this also holds for LTTS, we propose an evaluation scheme to assess potential emergent abilities in TTS, identifying seven categories that are challenging from the literature [27–31]: compound nouns, emotions, foreign words, paralinguistics, punctuations, questions, and syntactic complexities.”
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS (2021, methodology): “However, such approaches face challenges on handling alignment between phonemes and grapheme-based tokens, often require using word-level embeddings (thus a large vocabulary but still with out-of-vocabulary cases) [18, 19], or discarding a portion of tokens when subword embeddings are used [20].”
• PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “BERT, in particular, has proven effective in improving the performance of TTS models at either word level[4], character level [5, 6], or sentence level [7].”
• A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, unknown): “[18] include small versions of BERT in an RNN-based TTS system and highlight the importance of fine-tuning the pretrained model.”
• Acoustic Word Embeddings for End-to-End Speech Synthesis (2021, background): “[12] tries to fine-tune the BERT parameters with a prosody prediction task but still freezes the word piece embeddings.”
• Dependency Parsing based Semantic Representation Learning with Graph Neural Network for Enhancing Expressiveness of Text-to-Speech (2021, methodology): “Along different lines, CHiVE-BERT [16] incorporates a BERT model in an RNN-based speech synthesis model.”
• Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosody (2023, background): “BERT embeddings have also been sometimes incorporated into TTS [22].”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, methodology): “Recent works [9, 16, 17, 19] have applied the BERT model as an auxiliary encoder for the TTS system.”
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, methodology): “We further performed several additional direct comparisons using the diverse evaluation sets from [29].”