Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis

By Xin Eric Wang, Shinji Takaki, Junichi Yamagishi. 2019

138 citations recorded by the source corpus.

Neural waveform models have demonstrated better performance than conventional vocoders for statistical parametric speech synthesis. One of the best models, called WaveNet, uses an autoregressive (AR) approach to model the distribution of waveform sampling points, but it has to generate a waveform in a time-consuming sequential manner. Some new models that use inverse-autoregressive flow (IAF) can generate a whole waveform in a one-shot manner but require either a larger amount of training time or a complicated model architecture plus a blend of training criteria. As an alternative to AR and IAF-based frameworks, we propose a neural source-filter (NSF) waveform modeling framework that is straightforward to train and fast to generate waveforms. This framework requires three components to generate waveforms: a source module that generates a sine-based signal as excitation, a non-AR dilated-convolution-based filter module that transforms the excitation into a waveform, and a conditional module that pre-processes the input acoustic features for the source and filter modules. This framework minimizes spectral-amplitude distances for model training, which can be efficiently implemented using short-time Fourier transform routines. As an initial NSF study, we designed three NSF models under the proposed framework and compared them with WaveNet using our deep learning toolkit. It was demonstrated that the NSF models generated waveforms at least 100 times faster than our WaveNet-vocoder, and the quality of the synthetic speech from the best NSF model was comparable to that from WaveNet on a large single-speaker Japanese speech corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Yu Zhang
7. Heiga Zen
8. Haizhou Li

In Natural Language Processing

1. Junichi Yamagishi
2. Tao Qin
3. Tie-Yan Liu
4. Xu Tan
5. Heiga Zen
6. Zhenhua Ling
7. Yu Zhang
8. Tomoki Toda

In Speech & Audio

1. Junichi Yamagishi
2. Xu Tan
3. Tao Qin
4. Tie-Yan Liu
5. Frank K. Soong
6. Tomoki Toda
7. Yu Zhang
8. Xin Eric Wang

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Junichi Yamagishi
4. Tie-Yan Liu
5. Frank K. Soong
6. Tomoki Toda
7. Heiga Zen
8. Haizhou Li

Citations in the corpus

• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “, 2020), and the speech production mechanism (Juvela et al., 2019; Wang et al., 2020).”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “Another important type of vocoders is neural source-filter model [9, 10] based on the mechanism of human voice production.”
• A Survey on Neural Speech Synthesis (2021, background): “Other Vocoders Some works leverage neural-based source-filter model for waveform generation [375, 374, 371, 207, 146, 145, 75, 305, 407], aiming to achieve high voice quality while maintaining controllable speech generation.”
• DDSP: Differentiable Digital Signal Processing (2020, background): “Perhaps closest to this work, promising speech synthesis results were recently achieved using a differentiable waveshaping synthesizer (Wang et al., 2019).”
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, methodology): “Other approaches make strong assumptions about the signal in order to simplify the generation process (Engel et al., 2019; Wang et al., 2019), allowing real-time synthesis on CPU, while restricting the range of audio signals that can be modeled.”
• NAUTILUS: A Versatile Voice Cloning System (2020, methodology): “[16], is also a staple of many high-quality speech-generation recipes [9], [17].”
• Exploring TTS without T Using Biologically/Psychologically Motivated Neural Network Modules (ZeroSpeech 2020) (2020, methodology): “Figure 3: The neural source-filter model, which customized the hn-NSF model in [21].”
• A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis (2019, background): “for the harmonic and noise components of waveforms [43] was not adopted here.”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “faster than WaveNet vocoder, and yet achieving comparable voice quality on a large speech corpus [108].”
• Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN (2021, background): “For example, neural source-filter (NSF) [21, 22] realizes speech generation based on non-autoregressive modeling by non-linear filtering of parametrically generated source excitation signals with multiple dilated convolutional layers.”
• Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent Dilated Convolution Model for Parametric Speech Generation (2020, background): “Since the pitch controllability is an essential feature for a vocoder, NN-based SS models with carefully designed periodic and aperiodic inputs [24–26] greatly improve the pitch modeling capability.”
• Noise Level Limited Sub-Modeling for Diffusion Probabilistic Vocoders (2021, methodology): “Additionally, a signal-processing-based method, the neural source-filter [23], has been presented.”
• SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping (2022, background): “The ideas from classic vocoders should be useful [19, 20] because SpecGrad can be viewed as a classic vocoder with noise excitation.”
• Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders (2020, background): “Recently, some improved neural vocoders, such as glottal neural vocoder [21, 22], LPCNet [23], and neural source-filter (NSF) vocoder [24, 25, 26, 27], have been further proposed by combining speech production mechanisms with neural networks and have also demonstrated impressive performance.”
• HooliGAN: Robust, High Quality Neural Vocoding (2020, methodology): “NSF [2, 3] comprises of two excitation sources, namely a fundamental pitch (F0) driven sinusoidal signal and a constant Gaussian noise signal, each of which are then gated by an unvoiced/voiced (UV) switch, filtered by a simplified WaveNet (which they refer to as a neural filter) and passed through ”
• Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis (2019, methodology): “Note that the number of parameters WaveNet was more than that of the h-NSF models [15].”
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, methodology): “For the reconstruction loss, we used multi-scale spectrogram (MSS) loss (Wang et al., 2019) and mel spectrogram loss (Kong et al.”