Speaker and language factorization in DNN-based TTS synthesis

By Yuchen Fan, Yao Qian, Frank K. Soong, Lei He. 2016

31 citations recorded by the source corpus.

We have successfully proposed to use multi-speaker modelling in DNN-based TTS synthesis for improved voice quality with limited available data from a speaker. In this paper, we propose a new speaker and language factorized DNN, where speaker-specific layers are used for multi-speaker modelling, and shared layers and language-specific layers are employed for multi-language, linguistic feature transformation. Experimental results on a speech corpus of multiple speakers in both Mandarin and English show that the proposed factorized DNN can not only achieve a similar voice quality as that of a multi-speaker DNN, but also perform polyglot synthesis with a monolingual speaker's voice.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Sheng Zhao
7. Lior Wolf
8. Eliya Nachmani

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Lior Wolf
7. Eliya Nachmani
8. Jingzhou Yang

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “, a lot of works in statistic parametric speech synthesis have studied voice adaptation [79, 392, 450, 80, 67, 125], and the recent voice cloning challenge also attracts a lot of participants [394, 121, 337, 46].”
• Towards Universal Text-to-Speech (2020, methodology): “In [4], a factorized multilingual deep neural network (DNN) model has been proposed, where speakers and languages are factorized by using language and speaker-specific layers in the DNN.”
• Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data (2020, background): “In [2, 3], the authors model English-Chinese TTS with data from one and three bilingual speakers, in [3] one particular case of crosslingual transfer is shown.”
• Speaker Adaptation in DNN-Based Speech Synthesis Using d-Vectors (2017, methodology): “Recently, there has been increased effort to perform adaptation in deep neural network (DNN) based SPSS (DNN-TTS) [8, 9, 10, 11, 12, 13].”
• Unsupervised Polyglot Text-to-speech (2019, background): “In [26], a neural multilingual TTS is proposed.”
• Speaker Adaptation of a Multilingual Acoustic Model for Cross-Language Synthesis (2020, methodology): “Even though similar techniques have been proposed for DNN-based speech synthesis [15, 17, 16], most works in the literature have focused on monolingual systems that can generate multiple voices [22, 23, 24, 25].”