Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks

By Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari. 2017

213 citations recorded by the source corpus.

A method for statistical parametric speech synthesis incorporating generative adversarial networks (GANs) is proposed. Although powerful deep neural networks techniques can be applied to artificially synthesize speech waveform, the synthetic speech quality is low compared with that of natural speech. One of the issues causing the quality degradation is an oversmoothing effect often observed in the generated speech parameters. A GAN introduced in this paper consists of two neural networks: a discriminator to distinguish natural and generated samples, and a generator to deceive the discriminator. In the proposed framework incorporating the GANs, the discriminator is trained to distinguish natural and generated speech parameters, while the acoustic models are trained to minimize the weighted sum of the conventional minimum generation loss and an adversarial loss for deceiving the discriminator. Since the objective of the GANs is to minimize the divergence (i.e., distribution difference) between the natural and generated speech parameters, the proposed method effectively alleviates the oversmoothing effect on the generated speech parameters. We evaluated the effectiveness for text-to-speech and voice conversion, and found that the proposed method can generate more natural spectral parameters and $F_0$ than conventional minimum generation error training algorithm regardless of its hyperparameter settings. Furthermore, we investigated the effect of the divergence of various GANs, and found that a Wasserstein GAN minimizing the Earth-Mover's distance works the best in terms of improving the synthetic speech quality.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, background): “…the given spectrogram is real data or is generated by the inverter, which generates a realistic spectrogram to deceive the discriminator [14, 15, 16] After we deﬁne the multiple objectives for training, we update each module parameter θ C 2 S and θ Disc with the following equation: where…”
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, methodology): “The performance of the speech synthesis systems has been also significantly improved by implanting the GAN structure to the acoustic models [19–22], the postfilters [23], and the glottal excitations [24, 25].”
A New End-to-End Long-Time Speech Synthesis System Based on Tacotron2 (2019, unknown): “[13] Saito, Yuki, Shinnosuke Takamichi, and Hiroshi Saruwatari.”
StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks (2018, methodology): “While they have been found to be effective for use with image generation, in recent years they have also been employed with notable success for various speech processing tasks [7, 34–38].”
A New GAN-based End-to-End TTS Training Algorithm (2019, background): “There have been some studies on GAN in TTS in the past two years, such as GAN-based post ﬁlter [22], and GAN-based multi-task for TTS [23, 24].”
A Survey of Deep Learning: Platforms, Applications and Emerging Research Trends (2018, methodology): “[113] utilized GANs for statistical parametric speech synthesis to alleviate common over-smoothing effects.”
Conditional Spoken Digit Generation with StyleGAN (2020, methodology): “For the acoustic modeling task, adversarial training has been used as an auxiliary objective alongside conventional regression [8, 9] or as a generative post-filter to add stochasticity to regression-based predictions [10].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have been used for speech and audio analysis, such as synthesis [291], [292], [293], enhancement [294], and recognition [295].”
Wasserstein GAN and Waveform Loss-Based Acoustic Model Training for Multi-Speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder (2018, methodology): “In the framework proposed by [10], the generator acts as an acoustic model and is optimized by not only the conventional MSE but also an adversarial loss computed using the discriminator.”
Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion (2018, methodology): “GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
WaveNet Vocoder with Limited Training Data for Voice Conversion (2018, methodology): “Some methods tried to develop novel training criteria, such as minimizing sequence errors (SE) [11] and generative adversarial networks (GAN) [12].”
Singing Voice Synthesis Based on Generative Adversarial Networks (2019, methodology): “In text-to-speech synthesis, a GAN-based training method has been proposed [3, 17].”
Phase Reconstruction from Amplitude Spectrograms Based on Von-Mises-Distribution Deep Neural Network (2018, background): “There are two types of distributions: non-parametric [6, 7, 8, 9] and parametric ones.”