Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement

By Dongyang Dai, Li Chen, Yuping Wang, Mu Wang, Rui Xia, Xuchen Song, Zhiyong Wu, Yuxuan Wang. 2020

7 citations recorded by the source corpus.

With the popularity of deep neural network, speech synthesis task has achieved significant improvements based on the end-to-end encoder-decoder framework in the recent days. More and more applications relying on speech synthesis technology have been widely used in our daily life. Robust speech synthesis model depends on high quality and customized data which needs lots of collecting efforts. It is worth investigating how to take advantage of low-quality and low resource voice data which can be easily obtained from the Internet for usage of synthesizing personalized voice. In this paper, the proposed end-to-end speech synthesis model uses both speaker embedding and noise representation as conditional inputs to model speaker and noise information respectively. Firstly, the speech synthesis model is pre-trained with both multi-speaker clean data and noisy augmented data; then the pre-trained model is adapted on noisy low-resource new speaker data; finally, by setting the clean speech condition, the model can synthesize the new speaker's clean voice. Experimental results show that the speech generated by the proposed approach has better subjective evaluation results than the method directly fine-tuning pre-trained multi-speaker speech synthesis model with denoised new speaker data.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Sheng Zhao
6. Jinglin Liu
7. Frank K. Soong
8. Zhou Zhao

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Jinglin Liu
6. Sheng Zhao
7. Frank K. Soong
8. Zhou Zhao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Techniques Data Work Self-supervised Training Unpaired text or speech [51, 368, 433, 78, 140, 346, 197, 352, 71] Cross-lingual Transfer Paired text and speech [42, 390, 12, 60, 271, 105] Cross-speaker Transfer Paired text and speech [216, 125, 59, 39] Speech chain/Back transformation Unpaired text o”
• Denoispeech: Denoising Text to Speech with Frame-Level Noise Modeling (2020, methodology): “A straight-forward way [1, 2, 3, 4] is to first de-noise the noisy speech with a pre-trained speech enhancement model [5], and then use the enhanced speech to train the TTS model.”