Semi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis

By Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang, R. Skerry-Ryan. 2018

121 citations recorded by the source corpus.

Although end-to-end text-to-speech (TTS) models such as Tacotron have shown excellent results, they typically require a sizable set of high-quality pairs for training, which are expensive to collect. In this paper, we propose a semi-supervised training framework to improve the data efficiency of Tacotron. The idea is to allow Tacotron to utilize textual and acoustic knowledge contained in large, publicly-available text and speech corpora. Importantly, these external data are unpaired and potentially noisy. Specifically, first we embed each word in the input text into word vectors and condition the Tacotron encoder on them. We then use an unpaired speech corpus to pre-train the Tacotron decoder in the acoustic domain. Finally, we fine-tune the model using available paired data. We demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Wei-Ning Hsu
6. Hung-yi Lee
7. Sheng Zhao
8. James R. Glass

In Speech & Audio

1. Xu Tan
2. Hung-yi Lee
3. Wei-Ning Hsu
4. Tao Qin
5. Sheng Zhao
6. Tie-Yan Liu
7. Ye Jia
8. Yu Zhang

Citations in the corpus

• An Unsupervised Autoregressive Model for Speech Representation Learning (2019, methodology): “The model is optimized by minimizing the L1 loss (as is done when predicting spectral frames in some speech synthesis models [25, 26].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Self-supervised pretraining methods can be leveraged to enhance the language understanding or speech generation capabilities [51, 368, 433, 78].”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “…toward more challenging tasks, such as creating multi-speaker TTS systems [6–8], building neural end-to-end TTS systems from small amounts of data [9], utilizing a small amount of data for voice adaptation [8,10,11], investigating unsupervised prosody and speaking style modeling [12,13], and…”
• Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “Below we provide a brief overview of the self-supervised audio representations that are essential for SPEAR-TTS.”
• Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, background): “Finally, we are interested in exploring the usage of APC in other speech applications such as speech synthesis, where pre-training and transfer learning have already achieved some success [33, 34, 35, 36, 37].”
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS (2021, methodology): “However, such approaches face challenges on handling alignment between phonemes and grapheme-based tokens, often require using word-level embeddings (thus a large vocabulary but still with out-of-vocabulary cases) [18, 19], or discarding a portion of tokens when subword embeddings are used [20].”
• Cross-Lingual, Multi-Speaker Text-To-Speech Synthesis Using Neural Speaker Embedding (2019, methodology): “According to [7], it concludes that around 10 hours of speech-transcript pairs from one speaker are needed to get high quality by a neural end-to-end TTS model such as Tacotron.”
• End-to-End Code-Switching TTS with Cross-Lingual Language Model (2020, background): “A recent study on semi-supervised training [14] of Tacotron benefits from textual and acoustic knowledge obtained from large, publicly available text and speech corpora.”
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, background): “Chung et al. [20] perform semi-supervised training on Tacotron [3] in a pretrain-finetune manner.”
• AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN (2020, background): “However, building TTS system with limited data often sac-rifices quality and reliability [12].”
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement (2020, methodology): “Different techniques have been proposed for pre-training parts of a TTS model as a pre-training step and then fine-tuning it towards new speakers [4].”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “TL methods improve the synthesis quality of low-resource target voices [11, 13].”
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset (2021, background): “vised [15], semi-supervised [16], and cross-lingual transfer learning [17] have been developed.”
• Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “• Many seq2seq methods introduce phoneme input [14–16, 19, 23–35] rather than character input [10–22].”