Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training

By Jian Cong, Shan Yang, Lei Xie, Guoqiao Yu, Guanglu Wan. 2020

33 citations recorded by the source corpus.

Data efficient voice cloning aims at synthesizing target speaker's voice with only a few enrollment samples at hand. To this end, speaker adaptation and speaker encoding are two typical methods based on base model trained from multiple speakers. The former uses a small set of target speaker data to transfer the multi-speaker model to target speaker's voice through direct model update, while in the latter, only a few seconds of target speaker's audio directly goes through an extra speaker encoding model along with the multi-speaker model to synthesize target speaker's voice without model update. Nevertheless, the two methods need clean target speaker data. However, the samples provided by user may inevitably contain acoustic noise in real applications. It's still challenging to generating target voice with noisy data. In this paper, we study the data efficient voice cloning problem from noisy samples under the sequence-to-sequence based TTS paradigm. Specifically, we introduce domain adversarial training (DAT) to speaker adaptation and speaker encoding, which aims to disentangle noise from speech-noise mixture. Experiments show that for both speaker adaptation and encoding, the proposed approaches can consistently synthesize clean speech from noisy speaker samples, apparently outperforming the method adopting state-of-the-art speech enhancement module.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Yi Ren
7. Zhou Zhao
8. Lei He

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Sheng Zhao
6. Yi Ren
7. Zhou Zhao
8. Lei He

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “General Adaptation Modeling Variation Information [39] Increasing Data Coverage [56] Cross-Acoustic Adaptation [39, 53] Cross-Style Adaptation [398, 260, 120] Cross-Lingual Adaptation [438, 37, 206]”