Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions

By Dipjyoti Paul, Yannis Pantazis, Yannis Stylianou. 2020

34 citations recorded by the source corpus.

Recent advancements in deep learning led to human-level performance in single-speaker speech synthesis. However, there are still limitations in terms of speech quality when generalizing those systems into multiple-speaker models especially for unseen speakers and unseen recording qualities. For instance, conventional neural vocoders are adjusted to the training speaker and have poor generalization capabilities to unseen speakers. In this work, we propose a variant of WaveRNN, referred to as speaker conditional WaveRNN (SC-WaveRNN). We target towards the development of an efficient universal vocoder even for unseen speakers and recording conditions. In contrast to standard WaveRNN, SC-WaveRNN exploits additional information given in the form of speaker embeddings. Using publicly-available data for training, SC-WaveRNN achieves significantly better performance over baseline WaveRNN on both subjective and objective metrics. In MOS, SC-WaveRNN achieves an improvement of about 23% for seen speaker and seen recording condition and up to 95% for unseen speaker and unseen condition. Finally, we extend our work by implementing a multi-speaker text-to-speech (TTS) synthesis similar to zero-shot speaker adaptation. In terms of performance, our system has been preferred over the baseline TTS system by 60% over 15.5% and by 60.9% over 32.6%, for seen and unseen speakers, respectively.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Jinglin Liu
7. Rongjie Huang
8. Frank K. Soong

In Signal Processing

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Tao Qin
5. Tie-Yan Liu
6. Jinglin Liu
7. Rongjie Huang
8. Frank K. Soong

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “WaveRNN [209] Mel-Spectrogram AR / RNN SC-WaveRNN [259] Mel-Spectrogram AR / RNN MB WaveRNN [411] Mel-Spectrogram AR / RNN FFTNet [142] Cepstrum AR / CNN Par.”
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “We also include a comparison with SC-WaveRNN [37], a universal neural vocoder based on WaveRNN, using its official implementation.”
• SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model (2021, methodology): “Following the proposal of [2], [21] and [3] we use local sensitive attention [6].”
• The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “Among the AR vocoders, 4 teams used LPCNet [27] and one team used SC-WaveRNN [28] in their implementations.”
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus (2021, unknown): “Speaker Conditional WaveRNN [28] takes Session 29: Music, Speech and Audio Processing in Multimedia MM ’21, October 20–24, 2021, Virtual Event, China”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “A popular approach (Jia et al., 2018; Paul et al., 2020) to improve the robustness of the TTS model is to pre-train on a larger dataset consisting of various speeches to expand data distribution.”
• Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains (2020, unknown): “The robustness of WaveRNN using speaker representations has been discussed[19].”