Unsupervised Polyglot Text-to-speech
By Eliya Nachmani, Lior Wolf. 2019
43 citations recorded by the source corpus.
We present a TTS neural network that is able to produce speech in multiple languages. The proposed network is able to transfer a voice, which was presented as a sample in a source language, into one of several target languages. Training is done without using matching or parallel data, i.e., without samples of the same speaker in multiple languages, making the method much more applicable. The conversion is based on learning a polyglot network that has multiple per-language sub-networks and adding loss terms that preserve the speaker’s identity in multiple languages. We evaluate the proposed polyglot neural network for three languages with a total of more than 400 speakers and demonstrate convincing conversion capabilities.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “For example, based on the multi-speaker TTS model, Nachmani and Wolf [2019] introduce multiple encoders for each language and additional loss to keep the speaker’s identity.”
- Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis (2021, background): “Previous work about cross-lingual TTS usually trains a multilingual multi-speaker TTS model on multiple monolingual datasets by conditioning the TTS model on speaker and language representations called speaker and language embeddings, respectively [1, 2, 3, 4, 5].”
- Data-Efficient Training Strategies for Neural TTS Systems (2020, background): “Previous works [9, 22] aim to transfer voices across languages.”
- Towards Universal Text-to-Speech (2020, methodology): “Previous work [15] extends a multi-speaker system [8] to multilingual by using multiple language-depended speaker encoders.”
- Efficient neural speech synthesis for low-resource languages through multilingual modeling (2020, methodology): “First, we incorporated a separate encoder for each language to disentangle language information, similar to [11].”
- Speaker Adaptation of a Multilingual Acoustic Model for Cross-Language Synthesis (2020, background): “, synthesis in a language different from the target speaker’s language, where both languages are included in the training data) [12, 11, 13].”
- Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning (2019, background): “Most recently, [17] presented amultilingual neural TTSmodel which supports voice cloning across English, Spanish, and German.”
- One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech (2020, background): “[16] Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop,” in International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, Apr. 2018.”
- Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, methodology): “Therefore, unlike previous works that require manual design of language-specific and - agnostic modules such as per-language encoders (Nachmani & Wolf, 2019), we show that an architecture of multiple language-specific sub-networks with partially shared parameters may emerge from the multilingual…”
- Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training (2022, methodology): “This phenomenon has been exhibited in the previous work [9, 10, 12].”
- Open-source Multi-speaker Speech Corpora for Building Gujarati, Kannada, Malayalam, Marathi, Tamil and Telugu Speech Synthesis Systems (2020, background): “…the found data (Cooper, 2019; Baljekar, 2018), adaptation of the existing corpora to TTS needs (Zen et al., 2019) and development of novel techniques exploiting multilingual sharing, such as transfer learning (Baljekar et al., 2018; Chen et al., 2019; Nachmani and Wolf, 2019; Prakash et al., 2019).”
- Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, background): “Nachmani et al. [12] used consistency loss to remove the domain shift by minimizing the L1 distance between speaker embed-dings of source and target languages of one speaker.”
- Unsupervised Singing Voice Conversion (2019, methodology): “The similarity of the generated voices to the target voices is also evaluated automatically, using the same type of classification network that was used to verify voice identifiability in previous work [30, 31, 32, 33].”
- Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion (2020, methodology): “[10] presents a multi-lingual and multi-speaker neural TTS model based on the VoiceLoop architecture [11] with speaker and language embedding networks.”