Improving Unsupervised Style Transfer in end-to-end Speech Synthesis with end-to-end Speech Recognition

By Da-Rong Liu, Chi-Yu Yang, Szu-Lin Wu, Hung-yi Lee. 2018

18 citations recorded by the source corpus.

End-to-end TTS model can directly take an utterance as reference, and generate speech from the text with prosody and speaker characteristics similar to the reference utterance. Ideally, the transcription of reference utterance does not need to match the text to be synthesized, so unsupervised style transfer can be achieved. However, in the previous model, because only the matched text and speech are used in training, given unmatched text and speech during testing would make the model synthesize blurry speech. In this paper, we propose to mitigate the problem by using the unmatched text and speech during training, and using the ASR accuracy of an end-to-end ASR model to guide the training procedure. The experimental results show that with the guidance of end-to-end ASR, both the ASR accuracy (objective evaluation) and the listener preference (subjective evaluation) of the speech generated by TTS model are improved. Moreover, we propose attention consistency loss as regularization, which is shown to accelerate the training.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Sheng Zhao
7. Hisashi Kawai
8. Takuma Okamoto

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Hisashi Kawai
7. Takuma Okamoto
8. Yoshinori Shiga

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Technique Description Work Disentangling with Adversarial Training Disentanglement for control [218, 117, 275, 427] Cycle Consistency/Feedback for Control Enhance style/timbre generation [198, 380, 202, 30, 191] Semi-Supervised Learning for Control Use VAE and adversarial training [100, 116, 117, 42”
• Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation (2021, methodology): “It consists of both TTS and ASR, which are trained jointly to improve each other.”
• Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “• Many seq2seq methods introduce phoneme input [14–16, 19, 23–35] rather than character input [10–22].”
• Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models (2019, methodology): “The approach of joint learning has been tested in the literature in other domains than NLG/NLU for tasks such machine translation (Cheng et al., 2016; He et al., 2016; Tu et al., 2017) and speech processing (Tjandra et al., 2017, 2018; Liu et al., 2018).”