Towards Universal Text-to-Speech

By Jingzhou Yang, Lei He. 2020

30 citations recorded by the source corpus.

This paper studies a multilingual sequence-to-sequence text-to-speech framework towards universal modeling, that is able to synthesize speech for any speaker in any language using a single model. This framework consists of a transformer-based acoustic predictor and a WaveNet neural vocoder, with global conditions from speaker and language networks. It is examined on a massive TTS data set with around 1250 hours of data from 50 language locales, and the amount of data in different locales is highly unbalanced. Although the multilingual model exhibits the transfer learning ability to benefit the low-resource languages, data imbalance still undermines the model performance. A data balance training strategy is successfully applied and effectively improves the voice quality of the low-resource languages. Furthermore, this paper examines the modeling capacity of extending to new speakers and languages, as a key step towards universal modeling. Experiments show 20 seconds of data is feasible for a new speaker and 6 minutes for a new language.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Lei He
5. Frank K. Soong
6. Sheng Zhao
7. J. Yang
8. Jingzhou Yang

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Previous methods to tackle these challenges typically augment end-to-end TTS models with specific subnets for speaker and language control [Nachmani and Wolf, 2019, Zhang et al., 2020, Yang and He, 2020, Ellinas et al., 2022, Cai et al., 2023].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Cross-Lingual Transfer LRSpeech [396], [43, 12, 62, 407, 277, 108]”
• Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training (2022, methodology): “This phenomenon has been exhibited in the previous work [9, 10, 12].”
• Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, methodology): “We follow the approach in Yang & He (2020) to add language embeddings and speaker embeddings from the corresponding MLPs after the encoder.”