Deep Learning for Mandarin-Tibetan Cross-Lingual Speech Synthesis

By Weizhao Zhang, Hongwu Yang, Xiaolong Bu, Lili Wang. 2022

18 citations recorded by the source corpus.

This paper proposes a deep learning-based Mandarin-Tibetan cross-lingual speech synthesis to realize both Mandarin speech synthesis and Tibetan speech synthesis under a unique framework. Because Tibetan training corpus is hard to record, we train the acoustic models with a large scale Mandarin multi-speaker corpus and a small scale Tibetan one-speaker corpus. The acoustic models are trained with deep neural network (DNN), hybrid long short-term memory (LSTM), and hybrid bi-directional long short-term memory (BLSTM). We also further extend our Chinese text analyzer by adding a Tibetan text analyzer for generating context-dependent labels from input Chinese or Tibetan sentences. The Tibetan text analyzer includes a text normalization, a novel Tibetan word segmentation that combines a BLSTM with conditional random field, a prosodic boundary prediction, and a grapheme-to-phoneme conversion. We select the initials and the finals of both Mandarin and Tibetan as the speech synthesis units to train a speaker-independent mixed language average voice model (AVM) with DNN, hybrid LSTM, and hybrid BLSTM from Mandarin and Tibetan mixed corpus. Then the speaker adaptation is applied to train speaker-dependent DNN, hybrid LSTM, or hybrid BLSTM models of Mandarin or Tibetan with a small target speaker corpus from an AVM. Finally, we synthesize the Mandarin speech, or Tibetan speech though the speaker-dependent Mandarin or Tibetan models. The experiments show that the hybrid BLSTM-based cross-lingual speech synthesis framework is better than the other two cross-lingual frameworks and the Tibetan monolingual framework. The mixed Tibetan training corpus does not influence the voice quality of synthesized Mandarin speech. Furthermore, the hybrid BLSTM-based cross-lingual speech synthesis framework only needs 60% of the training corpus to synthesize a similar voice as the Tibetan monolingual framework. Therefore, the proposed method can be used for speech synthesis of low resource languages by borrowing the same tremendous resource language’s corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Sheng Zhao

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Zhaoxi Mu
6. Xinyu Yang
7. Yizhuo Dong
8. Sheng Zhao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Since human languages share similar vocal organs, pronunciations [383] and semantic structures [335], pre-training the TTS models on rich-resource languages can help the mapping between text and speech in low-resource languages [42, 390, 12, 60, 98, 336, 435, 241, 428].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[261] trained an average language model with a large Mandarin corpus and a small Tibetan corpus when training the Tibetan TTS model, which made up for the lack of Tibetan speech data.”