Wasserstein GAN and Waveform Loss-Based Acoustic Model Training for Multi-Speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder

By Yi Zhao, Shinji Takaki, Hieu-Thi Luong, Junichi Yamagishi, Daisuke Saito, Nobuaki Minematsu. 2018

67 citations recorded by the source corpus.

WaveNet, which learns directly from speech waveform samples, has been used as an alternative to vocoders and achieved very high-quality synthetic speech in terms of both naturalness and speaker similarity even in multi-speaker text-to-speech synthesis systems. However, the WaveNet vocoder uses acoustic features as local condition parameters, and these parameters need to be accurately predicted by another acoustic model. So far, it is not yet clear how to train this acoustic model, which is problematic because the final quality of synthetic speech is significantly affected by the performance of the acoustic model. Significant degradation occurs, especially when predicted acoustic features have mismatched characteristics compared to natural ones. In order to reduce the mismatched characteristics between natural and generated acoustic features, we propose new frameworks that incorporate either a conditional generative adversarial network (GAN) or its variant, Wasserstein GAN with gradient penalty (WGAN-GP), into multi-speaker speech synthesis that uses the WaveNet vocoder. The GAN generator performs as an acoustic model and its outputs are used as the local condition parameters of the WaveNet. We also extend the GAN frameworks and use the discretized-mixture-of-logistics (DML) loss of a well-trained WaveNet in addition to mean squared error and adversarial losses as parts of objective functions. Experimental results show that acoustic models trained using the WGAN-GP framework using back-propagated DML loss achieves the highest subjective evaluation scores in terms of both quality and speaker similarity.

Subfields: Artificial Intelligence, Physiology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Physiology

In Signal Processing

Citations in the corpus

Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, methodology): “The performance of the speech synthesis systems has been also significantly improved by implanting the GAN structure to the acoustic models [19–22], the postfilters [23], and the glottal excitations [24, 25].”
Conditional Spoken Digit Generation with StyleGAN (2020, methodology): “For the acoustic modeling task, adversarial training has been used as an auxiliary objective alongside conventional regression [8, 9] or as a generative post-filter to add stochasticity to regression-based predictions [10].”
WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN (2019, methodology): “The work closest to ours was the one proposed by [8], which uses a Wasserstein-GAN framework, followed by a WaveNet vocoder and a complimentary waveform based loss.”