Denoispeech: Denoising Text to Speech with Frame-Level Noise Modeling

By Chen Zhang, Yi Ren, Xu Tan, Jinglin Liu, Ke-jun Zhang, Tao Qin, Sheng Zhao, Tie-Yan Liu. 2020

50 citations recorded by the source corpus.

While neural-based text to speech (TTS) models can synthesize natural and intelligible voice, they usually require high-quality speech data, which is costly to collect. In many scenarios, only noisy speech of a target speaker is available, which presents challenges for TTS model training for this speaker. Previous works usually address the challenge using two methods: 1) training the TTS model using the speech denoised with an enhancement model; 2) taking a single noise embedding as input when training with noisy speech. However, they usually cannot handle speech with real-world complicated noise such as those with high variations along time. In this paper, we develop DenoiSpeech, a TTS system that can synthesize clean speech for a speaker with noisy speech data. In DenoiSpeech, we handle real-world noisy speech by modeling the fine-grained frame-level noise with a noise condition module, which is jointly trained with the TTS model. Experimental results on real-world data show that DenoiSpeech outperforms the previous two methods by 0.31 and 0.66 MOS respectively.1

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Zhou Zhao
5. Wei-Ning Hsu
6. Yi Ren
7. Yossi Adi
8. Juan Pino

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Zhou Zhao
4. Yi Ren
5. Jinglin Liu
6. Tie-Yan Liu
7. Rongjie Huang
8. Yossi Adi

Citations in the corpus

• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Text-to-speech (TTS) synthesis [53, 2, 49, 35, 48, 45, 29, 66, 43, 28] aims to generate human-like speech from text and has gained significant attention in the field of machine learning.”
• A Survey on Neural Speech Synthesis (2021, methodology): “[427] leverage gradient reversal or adversarial training to disentangle speaker timbre from noise in order to synthesize clean speech for noisy speakers.”
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, methodology): “Recent work (Zhang et al. 2021) trained the denoising module and the TTS system jointly with external noise data.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “However, such models usually require high-quality, and clean speech data (Zhang et al., 2021).”