Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora
By Yuewen Cao, Songxiang Liu, Xixin Wu, Shiyin Kang, Peng Liu, Zhiyong Wu, Xunying Liu, Dan Su, Dong Yu, Helen Mei-Ling Meng. 2020
22 citations recorded by the source corpus.
Synthesizing fluent code-switched (CS) speech with consistent voice using only monolingual corpora is still a challenging task, since language alternation seldom occurs during training and the speaker identity is directly correlated with language. In this paper, we present a bilingual phonetic posteriorgram (PPG) based CS speech synthesizer using only monolingual corpora. The bilingual PPG is used to bridge across speakers and languages, which is formed by stacking two monolingual PPGs extracted from two monolingual speaker-independent speech recognition systems. It is assumed that bilingual PPG can represent the articulation of speech sounds speaker-independently and captures accurate phonetic information of both languages in the same feature space. The proposed model first extracts bilingual PPGs from training data. Then an encoder- decoder based model is used to learn the relationship between input text and bilingual PPGs, and the bilingual PPGs are mapped to acoustic features using bidirectional long-short term memory based model conditioned on speaker embedding to control speaker identity. Experiments validate the effectiveness of the proposed model in terms of speech intelligibility, audio fidelity and speaker consistency of the generated code-switched speech.
Subfields: Artificial Intelligence, Experimental and Cognitive Psychology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shinnosuke Takamichi
- Hiroshi Saruwatari
- Detai Xin
- Tatsuya Komatsu
- Sakriani Watiasri Sakti
- Satoshi Nakamura
In Experimental and Cognitive Psychology
Citations in the corpus
- Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Code-Switch Speech Synthesis It is a common phenomenon to use code-switch utterances in bilingual or multi-lingual communities [Cao et al., 2020, Zhao et al., 2020, Manghat et al., 2022].”
- Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS (2021, background): “Cao et al. [14] used bilingual phonetic posteriorgrams as a language-independent feature to synthesize cross-lingual speech.”