Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models

By Wei Fang, Yu-An Chung, James R. Glass. 2019

29 citations recorded by the source corpus.

Modern text-to-speech (TTS) systems are able to generate audio that sounds almost as natural as human speech. However, the bar of developing high-quality TTS systems remains high since a sizable set of studio-quality pairs is usually required. Compared to commercial data used to develop state-of-the-art systems, publicly available data are usually worse in terms of both quality and size. Audio generated by TTS systems trained on publicly available data tends to not only sound less natural, but also exhibits more background noise. In this work, we aim to lower TTS systems' reliance on high-quality data by providing them the textual knowledge extracted by deep pre-trained language models during training. In particular, we investigate the use of BERT to assist the training of Tacotron-2, a state of the art TTS consisting of an encoder and an attention-based decoder. BERT representations learned from large amounts of unlabeled text data are shown to contain very rich semantic and syntactic information about the input text, and have potential to be leveraged by a TTS system to compensate the lack of high-quality data. We incorporate BERT as a parallel branch to the Tacotron-2 encoder with its own attention head. For an input text, it is simultaneously passed into BERT and the Tacotron-2 encoder. The representations extracted by the two branches are concatenated and then fed to the decoder. As a preliminary study, although we have not found incorporating BERT into Tacotron-2 generates more natural or cleaner speech at a human-perceivable level, we observe improvements in other aspects such as the model is being significantly better at knowing when to stop decoding such that there is much less babbling at the end of the synthesized audio and faster convergence during training.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Frank K. Soong
5. Lei He
6. Sheng Zhao
7. Jianfeng Gao
8. Mohit Bansal

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Lei He
5. Frank K. Soong
6. Laurent Besacier
7. Xiaodong He
8. Bowen Zhou

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “LowResource Self-Supervised Training [51, 368, 433, 78, 140, 346, 197, 352, 71]”
• A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “[14] enhance Tacotron2 by adding BERT as a secondary encoder.”
• Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis (2020, methodology): “More recently, efforts have been made to use additional text embeddings derived from pre-trained LMs, such as the BERT model [25], to improve the modelling of prosody [26–28].”
• Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS (2020, methodology): “[19][20] tried to use the contextual information extracted by Bert as an additional input to the Tacotron2 model.”
• Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input (2021, methodology): “non incremental) TTS have incorporated language model representations into neural TTS training and found that they could help speed up training time [14, 15].”
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement (2020, methodology): “We also computed the MCD of the training data during the transfer learning as explained in [16].”