Cross-Lingual, Multi-Speaker Text-To-Speech Synthesis Using Neural Speaker Embedding

By Mengnan Chen, Minchuan Chen, Shuang Liang, Jun Ma, Lei Chen, Shaojun Wang, Jing Xiao. 2019

61 citations recorded by the source corpus.

Neural network-based model for text-to-speech (TTS) synthesis has made significant progress in recent years. In this paper, we present a cross-lingual, multi-speaker neural end-to-end TTS framework which can model speaker characteristics and synthesize speech in different languages. We implement the model by introducing a separately trained neural speaker embedding network, which can represent the latent structure of different speakers and language pronunciations. We train the speech synthesis network bilingually and prove the possibility of synthesizing Chinese speaker’s English speech and vice versa. We explore different methods to fit a new speaker using only a few speech samples. The experimental results show that, with only several minutes of audio from a new speaker, the proposed model can synthesize speech bilingually and acquire decent naturalness and similarity for both languages.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Junichi Yamagishi
7. Sheng Zhao
8. Xin Eric Wang

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Lei He
7. Xin Eric Wang
8. Erica Cooper

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Adaptive General Adaptation AdaSpeech 1/3 [39, 398], [56, 260, 120, 37, 206]”
• Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “Several recent studies [9, 13, 14, 15] have used this approach for speaker modeling in TTS, with [15] modeling both speaker and language characteristics.”
• One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech (2020, methodology): “[17] used a phoneme-based Tacotron 2 with a ResCNN based speaker encoder [18] that enables a massively multi-speaker speech synthesis, even with fictitious voices.”
• Expressive Neural Voice Cloning (2021, background): “Recent efforts have been made in designing systems that can learn to synthesize a person’s voice from only a few audio samples [1, 2, 5, 9, 11].”
• Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, methodology): “Previous work reported that the synthesized cross-lingual speech suffered from naturalness and similarity degradation compared with the synthesized intra-lingual speech [8, 9].”
• Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions (2020, unknown): “[34, 35, 36].”
• Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training (2022, methodology): “Based on the encoder-decoder E2E framework, various multilingual text-to-speech (TTS) approaches have been proposed [7, 8, 9, 10, 11, 12, 13, 14].”
• Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “It can be further adapted in different tasks such as Speaker adaptation [7], cross-lingual Text-to-speech synthesis [8], and style conversion [9].”
• Multi-Lingual Multi-Speaker Text-to-Speech Synthesis for Voice Cloning with Online Speaker Enrollment (2020, background): “For example, [16] introduces a cross-lingual TTS system in English and Mandarin trained with IPA without language embedding.”
• Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers (2019, background): “For example, [16] introduces a cross-lingual TTS system in English and Mandarin trained with IPA without language embedding.”