One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech

By Tomás Nekvinda, Ondrej Dusek. 2020

62 citations recorded by the source corpus.

We introduce an approach to multilingual speech synthesis which uses the meta-learning concept of contextual parameter generation and produces natural-sounding multilingual speech using more languages and less training data than previous approaches. Our model is based on Tacotron 2 with a fully convolutional input text encoder whose weights are predicted by a separate parameter generator network. To boost voice cloning, the model uses an adversarial speaker classifier with a gradient reversal layer that removes speaker-specific information from the encoder. We arranged two experiments to compare our model with baselines using various levels of cross-lingual parameter sharing, in order to evaluate: (1) stability and performance when training on low amounts of data, (2) pronunciation accuracy and voice quality of code-switching synthesis. For training, we used the CSS10 dataset and our new small dataset based on Common Voice recordings in five languages. Our model is shown to effectively share information across languages and according to a subjective evaluation test, it produces more natural and accurate code-switching speech than the baselines.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Wei-Ning Hsu
5. Frank K. Soong
6. Lei He
7. Yossi Adi
8. Michael Auli

Citations in the corpus

• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, neutral): “In parallel with the ZS-TTS, multilingual TTS has also evolved aiming at learning models for multiple languages at the same time [14, 15, 16, 17].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Information Type Explicit Language/Style/Speaker ID [438, 241, 191, 158, 38] Pitch/Duration/Energy [284, 286, 177, 155, 233, 359]”
• Scaling Speech Technology to 1, 000+ Languages (2023, background): “Additionally, various modeling schemes have been developed to encourage knowledge sharing between languages, such as parameter generation networks [Nekvinda and Dušek, 2020] and leveraging unpaired speech or text data for pre-training [Saeki et al.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Nekvinda T, Dušek O (2020) One model, many languages: Meta-learning for multilingual text-tospeech. arXiv preprint arXiv:200800768 148.”
• Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, methodology): “Other multilingual methods leverage per-language encoder (Nachmani & Wolf, 2019), possibly with parameters predicted from language ID (Nekvinda & Dusek, 2020).”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “Since the lexical content is also constrained by the language, there have been some efforts to build a cross-lingual (Sisman et al., 2019a) or multi-lingual (Nekvinda and Dušek, 2020) speech synthesis system in current literature.”