Pre-Trained Text Embeddings for Enhanced Text-to-Speech Synthesis

By Tomoki Hayashi, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Shubham Toshniwal, Karen Livescu. 2019

76 citations recorded by the source corpus.

We propose an end-to-end text-to-speech (TTS) synthesis model that explicitly uses information from pre-trained embeddings of the text. Recent work in natural language processing has developed self-supervised representations of text that have proven very effective as pre-training for language understanding tasks. We propose using one such pre-trained representation (BERT) to encode input phrases, as an additional input to a Tacotron2-based sequence-to-sequence TTS model. We hypothesize that the text embeddings contain information about the semantics of the phrase and the importance of each word, which should help TTS systems produce more natural prosody and pronunciation. We conduct subjective listening tests of our proposed models using the 24-hour LJSpeech corpus, finding that they improve mean opinion scores modestly but significantly over a baseline TTS model without pre-trained text embedding input.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Lei He
7. James R. Glass
8. Chenpeng Du

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Sheng Zhao
7. Chenpeng Du
8. Yanqing Liu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “• Text pre-training [78, 101, 387, 140, 95, 447], which can provide better text representations by using pre-trained word embeddings or model parameters.”
• Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, background): “Finally, we are interested in exploring the usage of APC in other speech applications such as speech synthesis, where pre-training and transfer learning have already achieved some success [33, 34, 35, 36, 37].”
• Acoustic Word Embeddings for End-to-End Speech Synthesis (2021, methodology): “[10,11] extract hidden states of BERT and pass them to the TTS model.”
• Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model (2020, methodology): “In [11, 13], large BERT models are used: 12-layers, and 768 hidden units in each layer (in [12] the size of the BERT models are not specified).”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, methodology): “Recent works [9, 16, 17, 19] have applied the BERT model as an auxiliary encoder for the TTS system.”
• Dependency Parsing based Semantic Representation Learning with Graph Neural Network for Enhancing Expressiveness of Text-to-Speech (2021, background): “Pre-trained BERT is firstly introduced to Tacotron-2 as an additional input and show gains in mean opinion scores [14].”
• A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, background): “[15] compare subword-level and phrase-level representations from BERT-large as extra inputs to Tacotron2 decoder.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[70] directly used BERT as a context feature extraction network to encode input text, and added encoded word or sentencelevel contextual information to the input of the encoder of the acoustic model to improve the quality of synthesized speech.”
• End-to-End Code-Switching TTS with Cross-Lingual Language Model (2020, methodology): “In [15], Hayashi et al. show that, the text embeddings computed from a pre-trained BERT model, help TTS systems improve nat-uralness of generated speech.”
• Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis (2020, methodology): “More recently, efforts have been made to use additional text embeddings derived from pre-trained LMs, such as the BERT model [25], to improve the modelling of prosody [26–28].”
• Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS (2020, methodology): “[19][20] tried to use the contextual information extracted by Bert as an additional input to the Tacotron2 model.”
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, methodology): “[26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019.”