Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

By Detai Xin, Tatsuya Komatsu, Shinnosuke Takamichi, Hiroshi Saruwatari. 2021

31 citations recorded by the source corpus.

We propose a method for obtaining disentangled speaker and language representations via mutual information minimization and domain adaptation for cross-lingual text-to-speech (TTS) synthesis. The proposed method extracts speaker and language embeddings from acoustic features by a speaker encoder and a language encoder. Then the proposed method applies domain adaptation on the two embeddings to obtain language-invariant speaker embedding and speaker-invariant language embedding. To get more disentangled representations, the proposed method further uses mutual information minimization between the two embeddings to remove entangled information within each embedding. Disentangled representations of speaker and language are critical for cross-lingual TTS synthesis since entangled representations make it difficult to maintain speaker identity information when changing the language representation and consequently causes performance degradation. We evaluate the proposed method using English and Japanese multi-speaker datasets with a total of 207 speakers. Experimental result demonstrates that the proposed method significantly improves the naturalness and speaker similarity of both intra-lingual and cross-lingual TTS synthesis. Furthermore, we show that the proposed method has a good capability of maintaining the speaker identity between languages.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, neutral): “Some previous works [19, 20] apply speaker classification loss on generated mel-spectrograms as a supervision to ensure the synthesized speech to be more similar to reference speech, which, however, does not bring much improvement in speaker similarity [16].”
Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis (2021, methodology): “However, our previous work [1, 2] showed that when the speaker recognition model was trained on multilingual data, domain shift of different languages was easy to appear, which would impede extending the monolingual method to multilingual settings.”
Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “Other TTS models capable of cross-lingual voice modelling, such as Zhang et al. (2019 2021), could also be utilized.”