Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training
12 citations recorded by the source corpus.
In cross-lingual speech synthesis, the speech in various languages can be synthesized for a monoglot speaker. Normally, only the data of monoglot speakers are available for model training, thus the speaker similarity is relatively low between the synthesized cross-lingual speech and the native language recordings. Based on the multilingual transformer text-to-speech model, this paper studies a multi-task learning framework to improve the cross-lingual speaker similarity. To further improve the speaker similarity, joint training with a speaker classifier is proposed. Here, a scheme similar to parallel scheduled sampling is proposed to train the transformer model efficiently to avoid breaking the parallel training mechanism when introducing joint training. By using multi-task learning and speaker classifier joint training, in subjective and objective evaluations, the cross-lingual speaker similarity can be consistently improved for both the seen and unseen speakers in the training set.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Compared with the fixed speaker embedding extracted from the pretrained speaker encoder, a multi-task learning framework has been proposed to enhance cross-lingual speaker similarity by simultaneously training speaker classification (Yang and He, 2022).”
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Yang and He [2022] further propose a multi-task learning method with additional tasks of speaker similarity and language identification.”