WaveNet Vocoder with Limited Training Data for Voice Conversion

By Li-Juan Liu, Zhenhua Ling, Yuan Jiang, M. Zhou, Lirong Dai. 2018

127 citations recorded by the source corpus.

This paper investigates the approaches of building WaveNet vocoders with limited training data for voice conversion (VC). Current VC systems using statistical acoustic models always suffer from the quality degradation of converted speech. One of the major causes is the use of hand-crafted vocoders for wave-form generation. Recently, with the emergence of WaveNet for waveform modeling, speaker-dependent WaveNet vocoders have been proposed and they can reconstruct speech with better quality than conventional vocoders, such as STRAIGHT. Because training a WaveNet vocoder in the speaker-dependent way requires a relatively large training dataset, it remains a challenge to build a high-quality WaveNet vocoder for VC tasks when the training data of target speakers is limited. In this paper, we propose to build WaveNet vocoders by combining the initialization using a multi-speaker corpus and the adaptation using a small amount of target data, and evaluate this proposed method on the Voice Conversion Challenge (VCC) 2018 dataset which contains approximately 5 minute recordings for each target speaker. Experimental results show that the WaveNet vocoders built using our proposed method outperform conventional STRAIGHT vocoder. Furthermore, our system achieves an average naturalness MOS of 4.13 in VCC 2018, which is the highest among all submitted systems.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, background): “Recent DNN-based codecs [13][14][15][16] model speech signals in time domain directly without T-F transformation.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, background): “speakers [39], neural voice cloning [2, 13], voice conversion [26], speech-to-speech translation [12], and neural audio codec [59], where the studio-quality recordings are often not available.”
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, methodology): “Vocoders have been employed in various fields such as text-tospeech[1, 2, 3, 4], voice conversion[5], and speech-to-speech translation[6].”
ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech (2019, methodology): “These WaveNet vocoders (μ-law companding and 10-bit quantization) were trained by pre-training them using an in-house multi-speaker corpus and fine-tuning it using the speech of target speakers [38].”
Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions (2020, background): “Previous studies also attempted to improve adaptation capabilities of vocoders [24], either with or without providing speaker information [25, 26].”
Towards Achieving Robust Universal Neural Vocoding (2018, background): “Several recent studies attempted to improve the adaptation capabilities of such models [18, 19], commonly using explicit speaker information (either as a onehot encoding or some other form of speaker embedding) [20].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “[209] WaveNet vocoder MCC and F0 ▪ VCC 2018 ▪ Performance degrades on inter-gender conversions”
Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet (2019, background): “To give a few examples, the recent advanced high-performance VC systems gain from the use of the phone posteriorgram (that is, a continuous phone representation) of inputted speech [18].”
Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora (2020, methodology): “Beneﬁting from its speaker-independent property, PPGs have been successfully used in voice conversion (VC) tasks [12–14].”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “Details of best performing systems are reported in [274], [275].”
NAUTILUS: A Versatile Voice Cloning System (2020, methodology): “[16], is also a staple of many high-quality speech-generation recipes [9], [17].”
Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains (2020, unknown): “Recently, vocoders have been utilized in various fields such as text-to-speech[1, 2, 3, 4] and voice conversion[5] or speech-to-speech translation[6].”
Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders (2020, methodology): “Based on these waveform generation models, neural vocoders have been developed [11–16], which reconstruct speech waveforms from various acoustic features for SPSS, voice conversion [17, 18], bandwidth extension [19], etc.”
A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis (2019, methodology): “els [16]–[21] have been developed to reconstruct speech waveforms from various acoustic features for SPSS and some other tasks, such as voice conversion [22], [23], bandwidth exten-”
Robust universal neural vocoding (2018, background): “Several recent studies attempted to improve the adaptation capabilities of such models [15, 16], commonly using explicit speaker information (either as a onehot encoding or some other form of speaker embedding) [17].”
TTS Skins: Speaker Conversion via ASR (2019, methodology): “2 shows ours results versus the best performing method of the VCC 2018, which is known as N10 [1].”