Towards Achieving Robust Universal Neural Vocoding

By Jaime Lorenzo-Trueba, Thomas Drugman, Javier Latorre, Thomas Merritt, Bartosz Putrycz, Roberto Barra-Chicote, Alexis Moinet, Vatsal Aggarwal. 2018

96 citations recorded by the source corpus.

This paper explores the potential universality of neural vocoders. We train a WaveRNN-based vocoder on 74 speakers coming from 17 languages. This vocoder is shown to be capable of generating speech of consistently good quality (98% relative mean MUSHRA when compared to natural speech) regardless of whether the input spectrogram comes from a speaker or style seen during training or from an out-of-domain scenario when the recording conditions are studio-quality. When the recordings show significant changes in quality, or when moving towards non-speech vocalizations or singing, the vocoder still significantly outperforms speaker-dependent vocoders, but operates at a lower average relative MUSHRA of 75%. These results are shown to be consistent across languages, regardless of them being seen during training (e.g. English or Japanese) or unseen (e.g. Wolof, Swahili, Ahmaric).

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Hung-yi Lee
7. Sheng Zhao
8. Zhenhua Ling

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Hung-yi Lee
7. Zhenhua Ling
8. Yang Ai

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “WaveRNN [209] Mel-Spectrogram AR / RNN SC-WaveRNN [259] Mel-Spectrogram AR / RNN MB WaveRNN [411] Mel-Spectrogram AR / RNN FFTNet [142] Cepstrum AR / CNN Par.”
• Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, methodology): “Finally, the LPCNet model serves as another comparison point for a compression only neural codec.”
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “In previous work, WaveRNN has been applied for universal vocoding [28, 37].”
• Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders (2020, methodology): “Neural vocoders based on these waveform generation models [11, 12, 13, 14, 15, 16] have been developed to reconstruct speech waveforms from various acoustic features for SPSS and some other tasks, such”
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech (2020, methodology): “In recent times, neural text-to-speech (NTTS) methods have significantly improved the naturalness of synthesised speech obtained from TTS systems [1–8].”
• End-To-End Speech Synthesis Applied to Brazilian Portuguese (2020, methodology): “We have used the following implementations: DCTTS provided by Park [2018a]; WaveRNN provided by Wang [2018]; Universal WaverRNN [Lorenzo-Trueba et al., 2018] provided by Gölge [2019]; Tacotron 1 provided by Park [2018b]; Mozilla TTS, based on Tacotron 2, and provided by Mozilla [2019].”
• Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, methodology): “To reduce computational costs, we replace the WaveNet decoder [25] with a lightweight RNN-based vocoder [28].”
• Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions (2020, background): “In [27], a potential universal vocoder was introduced claiming that speaker encoding is not essential to train a high-quality neural vocoder.”
• Using Vaes and Normalizing Flows for One-Shot Text-To-Speech Synthesis of Expressive Speech (2019, unknown): “To complete the TTS pipeline, we use the vocoder of [16] to convert the mel-spectrograms to audio waveforms.”
• A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis (2019, methodology): “els [16]–[21] have been developed to reconstruct speech waveforms from various acoustic features for SPSS and some other tasks, such as voice conversion [22], [23], bandwidth exten-”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, unknown): “domain and out-of-domain spectrogram [101]–[103].”
• Towards Robust Neural Vocoding for Speech Generation: A Survey (2019, background): “Recently, [18] reported that a WaveRNN-based neural vocoder trained on multi-speaker multilingual data can generate natural speech despite conditions from an unseen domain.”
• Universal Neural Vocoding with Parallel Wavenet (2021, methodology): “In this section, we compare UPW with other popular neural vocoder architectures, including Universal WaveRNN (UWRNN) [19], Parallel WaveGAN (PWGAN) [11] and WaveGlow (WGlow) [9].”
• An Efficient Subband Linear Prediction for LPCNet-Based Neural Synthesis (2020, background): “Moreover, it also shows the robustness in several scenarios[6, 7].”
• Avocodo: Generative Adversarial Network for Artifact-free Vocoder (2022, background): “These artifacts significantly reduce the perceptual quality, when synthesizing speech with large pitch variation (Lorenzo-Trueba et al. 2019; Zaı̈di et al. 2022).”
• Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains (2020, unknown): “When training WaveGlow[16]2 and WaveRNN[10]3, GitHub implementations were used for reproducibility.”
• Nonparallel Expressive TTS for Unseen Target Speaker using Style-Controlled Adaptive Layer and Optimized Pitch Embedding (2023, methodology): “The predicted Mel-spectrograms are then synthesized into speech using a universal vocoder [26].”