Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning

By Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Z. Chen, R. Skerry-Ryan, Ye Jia, Andrew Rosenberg, Bhuvana Ramabhadran. 2019

205 citations recorded by the source corpus.

We present a multispeaker, multilingual text-to-speech (TTS) synthesis model based on Tacotron that is able to produce high quality speech in multiple languages. Moreover, the model is able to transfer voices across languages, e.g. synthesize fluent Spanish speech using an English speaker's voice, without training on any bilingual or parallel examples. Such transfer works across distantly related languages, e.g. English and Mandarin. Critical to achieving this result are: 1. using a phonemic input representation to encourage sharing of model capacity across languages, and 2. incorporating an adversarial loss term to encourage the model to disentangle its representation of speaker identity (which is perfectly correlated with language in the training data) from the speech content. Further scaling up the model by training on multiple speakers of each language, and incorporating an autoencoding input to help stabilize attention during training, results in a model which can be used to consistently synthesize intelligible speech for training speakers in all languages seen during training, and in native or foreign accents.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “However, the above methods fail to effectively extend to zero-shot scenarios for synthesizing target speech from the unseen source speaker, and often suffer from low speaker similarity and L2 (second-language, or foreign) accent problems [Zhang et al., 2019, Lee et al., 2022].”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, neutral): “In parallel with the ZS-TTS, multilingual TTS has also evolved aiming at learning models for multiple languages at the same time (Cao et al., 2019; Zhang et al., 2019; Nekvinda & Dušek, 2020; Li et al., 2021).”
A Survey on Neural Speech Synthesis (2021, methodology): “Information Type Explicit Language/Style/Speaker ID [438, 241, 191, 158, 38] Pitch/Duration/Energy [284, 286, 177, 155, 233, 359]”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, background): “Yet another way to address this problem is to augment the model by incorporating an auxiliary representation to capture the latent factors in the speech data [27–30].”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “…and “distortion”, which is likely the result of noisy reference speech from CoV-oST 2 used for voice transferring; the speaker similarity is largely impacted by “different languages”, which does not necessarily reﬂect voice difference (same as observed in Zhang et al. (2019); Jia et al. (2022)).”
Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora (2020, methodology): “The baseline approach follows a Tacotron2-based cross-lingual voice cloning model [28], where a speaker-adversial loss is incorporated to disentangle closely correlated textual content and speaker identity.”
Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data (2020, methodology): “Following [7], we also add a domain adversarial neural network (DANN) [13] to learn speaker-independent encoded features.”
Phonological Features for 0-shot Multilingual Speech Synthesis (2020, methodology): “Zhang and colleagues [1] first noted the ability of a multilingual, multi-speaker Tacotron 2 to code-switch, i.”
The STC text-to-speech system for Blizzard Challenge 2019 (2019, unknown): “Second, it is flexible: there is a lot of research into various add-ons and modifications to this architecture, aimed at improving overall quality and/or making some kind of prosody control feasible [25, 24, 26], improving stability [27], semi-supervised learning, transfer learning and efficient use”
Data-Efficient Training Strategies for Neural TTS Systems (2020, background): “Previous works [9, 22] aim to transfer voices across languages.”
Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training (2022, methodology): “In [9], a speaker classifier is introduced under the domain-adversarial training framework to encourage the encoded phone sequence to be speaker-independent.”
Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis (2021, background): “Previous work about cross-lingual TTS usually trains a multilingual multi-speaker TTS model on multiple monolingual datasets by conditioning the TTS model on speaker and language representations called speaker and language embeddings, respectively [1, 2, 3, 4, 5].”
One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech (2020, methodology): “[14] used Tacotron 2 [1] conditioned on phonemes and showed voicecloning abilities on English, Spanish, and Chinese.”
Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers (2019, methodology): “While we are preparing for this submission, we notice a new non-refereed publication from Google [17] on arXiv recently which shares many ideas in our system.”
AI-generated characters for supporting personalized learning and well-being (2021, methodology): “To synthesize the generated character’s speech, we use a state-of-the-art publicly available real-time voice cloning algorithm 5,67 that enables using a few seconds of reference audio to synthesize new speech in that speaker’s voice, using the text data.”
Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation (2022, methodology): “Various approaches can be applied on this task, such as standard TTS architectures [6] or even voice conversion [7].”