A Comparison of Recent Waveform Generation and Acoustic Modeling Methods for Neural-Network-Based Speech Synthesis

By Xin Wang, Jaime Lorenzo-Trueba, Shinji Takaki, Lauri Juvela, Junichi Yamagishi. 2018

69 citations recorded by the source corpus.

Recent advances in speech synthesis suggest that limitations such as the lossy nature of the amplitude spectrum with minimum phase approximation and the over-smoothing effect in acoustic modeling can be overcome by using advanced machine learning approaches. In this paper, we build a framework in which we can fairly compare new vocoding and acoustic modeling techniques with conventional approaches by means of a large scale crowdsourced evaluation. Results on acoustic models showed that generative adversarial networks and an autoregressive (AR) model performed better than a normal recurrent network and the AR model performed best. Evaluation on vocoders by using the same AR acoustic model demonstrated that a Wavenet vocoder outperformed classical source-filter-based vocoders. Particularly, generated speech waveforms from the combination of AR acoustic model and Wavenet vocoder achieved a similar score of speech quality to vocoded speech.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, background): “[373] conduct thorough experimental studies on different acoustic models.”
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, methodology): “The scales were [2, 2, 2, 3, 5].”
Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis (2019, methodology): “6This new strategy slightly improved the aperiodicity of voiced sounds, compared with the WaveNet in our previous study, which selected the most probable waveform value at all voiced steps [13].”
Neural Source-filter-based Waveform Model for Statistical Parametric Speech Synthesis (2018, methodology): “Each model generated waveforms using natural and generated acoustic features, where the generated acoustic features were produced by the acoustic models in our previous study [6].”
A Comparison of Recent Neural Vocoders for Speech Signal Reconstruction (2019, background): “In [6], neural vocoders (WaveNet) were included in the benchmark.”
Investigation of Enhanced Tacotron Text-to-speech Synthesis Systems with Self-attention for Pitch Accent Language (2018, methodology): “For baseline systems, we included two classical pipeline systems that use vocoder parameters and mel-spectrogram [16], [18], [19].”
ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech (2019, methodology): “The WaveNet vocoder followed the recipe in [19] but uses a Gaussian distribution to directly model continuousvalued raw waveforms.”
Towards Achieving Robust Universal Neural Vocoding (2018, background): “They allow us to overcome the shortcomings of traditional methods [16] at a very significant cost in computation power and data requirements.”
A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural $F_0$ Model for Statistical Parametric Speech Synthesis (2020, unknown): “structure and training recipe to the WaveNet vocoder in another of our studies [54].”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “learns to reconstruct an audio waveform from acoustic features [53].”
Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis (2019, methodology): “Although WaveNet outperformed traditional vocoders [9], its sequential generation process is prohibitively slow.”
Wasserstein GAN and Waveform Loss-Based Acoustic Model Training for Multi-Speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder (2018, background): “such as cepstrum, F0, or spectrograms only [14], and results have shown that the sound quality of the WaveNet vocoder outperformed deterministic vocoders and phase recovery algorithms [16].”
GELP: GAN-Excited Linear Prediction for Speech Synthesis from Mel-spectrogram (2019, methodology): “With this division of labor, neural vocoders have been shown to produce high-quality synthetic speech when paired with a sequence-tosequence model [8], or a more conventional SPSS pipeline [9].”
Speaker-independent raw waveform model for glottal excitation (2018, methodology): “The prediction of signal sample probability distributions allows manual adjustment of the sampling strategy at test time, for example, mode sampling in voiced regions has been reported to improve perceived synthetic speech quality [5].”
Robust universal neural vocoding (2018, background): “They allow us to overcome the shortcomings of traditional methods [13] at a very significant cost in computation power and data requirements.”
ExcitNet Vocoder: A Neural Excitation Model for Parametric Speech Synthesis Systems (2018, background): “This enables the system to automatically learn the relationship between acoustic features and speech samples which results in superior perceptual quality over traditional LPC vocoders [13], [24].”
GlotNet—A Raw Waveform Model for the Glottal Excitation in Statistical Parametric Speech Synthesis (2019, background): “Repetition upsampling also is commonly used [16], [17], but likewise exhibits discontinuities, while having virtually the same computational cost as linear interpolation.”
Improving FFTNet Vocoder with Noise Shaping and Subband Approaches (2018, methodology): “Neural vocoders based on WaveNet [28] and SampleRNN [29] have been applied to SPSS [30–34] and VC [35–39], and also outperform conventional source-ﬁlter vocoders.”