End-to-end Code-switched TTS with Mix of Monolingual Recordings

By Yuewen Cao, Xixin Wu, Songxiang Liu, Jianwei Yu, Xu Li, Zhiyong Wu, Xunying Liu, Helen Mei-Ling Meng. 2019

36 citations recorded by the source corpus.

State-of-the-art text-to-speech (TTS) synthesis models can produce monolingual speech with high intelligibility and naturalness. However, when the models are applied to synthesize code-switched (CS) speech, the performance declines seriously. Conventionally, developing a CS TTS system requires multilingual data to incorporate language-specific and cross-lingual knowledge. Recently, end-to-end (E2E) architecture has achieved satisfactory results in monolingual TTS. The architecture enables the training from one end of alphabetic text input to the other end of acoustic feature output. In this paper, we explore the use of E2E framework for CS TTS, using a combination of Mandarin and English monolingual speech corpus uttered by two female speakers. To handle alphabetic input from different languages, we explore two kinds of encoders: (1) shared multilingual encoder with explicit language embedding (LDE); (2) separated monolingual encoder (SPE) for each language. The two systems use identical decoder architecture, where a discriminative code is incorporated to enable the model to generate speech in one speaker’s voice consistently. Experiments confirm the effectiveness of the proposed modifications on the E2E TTS framework in terms of quality and speaker similarity of the generated speech. Moreover, our proposed systems can generate controllable foreign-accented speech at character-level using only mixture of monolingual training data.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, background): “In parallel with the ZS-TTS, multilingual TTS has also evolved aiming at learning models for multiple languages at the same time [15, 16, 17, 18].”
One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech (2020, methodology): “[19], respectively, but were slightly adapted to our tasks for a fairer comparison to GEN – we use more languages and less data than the original works.”
Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora (2020, background): “Separate encoder and shared encoder with language embedding are investigated in [25] to encode individual languages with characters as input.”
End-to-End Code-Switching TTS with Cross-Lingual Language Model (2020, background): “In the context of CS TTS [10], two kinds of encoders are explored to handle alphabetic inputs from different languages: shared multilingual encoder with explicit language embedding and separated monolingual encoder, both of which produce more natural CS utterances than Tacotron [11].”
Efficient neural speech synthesis for low-resource languages through multilingual modeling (2020, methodology): “More recently, multilingual modeling was also adopted in S2S architectures [11, 12, 13, 14, 15, 16], however mostly for the purposes of code-mixing and cross-lingual synthesis.”
Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “• Many seq2seq methods introduce phoneme input [14–16, 19, 23–35] rather than character input [10–22].”
Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion (2020, methodology): “[12] proposes code-switched TTS using Tacotron-based end-toend systems [13].”