Statistical parametric speech synthesis using generative adversarial networks under a multi-task learning framework

By Shan Yang, Lei Xie, Xiao Chen, Xiaoyan Lou, Xuan Zhu, Dongyan Huang, Haizhou Li. 2017

48 citations recorded by the source corpus.

In this paper, we aim at improving the performance of synthesized speech in statistical parametric speech synthesis (SPSS) based on a generative adversarial network (GAN). In particular, we propose a novel architecture combining the traditional acoustic loss function and the GAN's discriminative loss under a multi-task learning (MTL) framework. The mean squared error (MSE) is usually used to estimate the parameters of deep neural networks, which only considers the numerical difference between the raw audio and the synthesized one. To mitigate this problem, we introduce the GAN as a second task to determine if the input is a natural speech with specific conditions. In this MTL framework, the MSE optimization improves the stability of GAN, and at the same time GAN produces samples with a distribution closer to natural speech. Listening tests show that the multi-task architecture can generate more natural speech that satisfies human perception than the conventional methods.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Lei Xie
7. Sheng Zhao
8. Yi Ren

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Lei Xie
7. Yi Ren
8. Shan Yang

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “[402] leverage GAN [87] to improve the generation quality of acoustic features.”
• Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, methodology): “The performance of the speech synthesis systems has been also significantly improved by implanting the GAN structure to the acoustic models [19–22], the postfilters [23], and the glottal excitations [24, 25].”
• Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS (2020, methodology): “It has also been used in speech synthesis [19], such as statistical parametric speech synthesis with GANs [20] and DNN-based speech synthesis with stacked bottleneck features [21].”
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis (2021, methodology): “To overcome the above issue, generative adversarial networks are utilized to model the distribution of target speech [13, 14].”
• A New End-to-End Long-Time Speech Synthesis System Based on Tacotron2 (2019, background): “[14] Shan Yang, Lei Xie, Xiao Chen, Xiaoyan Lou, Xuan Zhu, Dongyan Huang, Haizhou Li, " Statistical Parametric Speech Synthesis Using Generative Adversarial Networks Under A Multi-task Learning Framework", arXiv:1707.”
• WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN (2019, methodology): “[9] use the mean squared error (MSE) and a variational autoencoder (VAE) to enable the GAN optimization process in a multi-task learning framework.”
• Singing Voice Synthesis Based on Generative Adversarial Networks (2019, methodology): “In text-to-speech synthesis, a GAN-based training method has been proposed [3, 17].”
• A New GAN-based End-to-End TTS Training Algorithm (2019, background): “There have been some studies on GAN in TTS in the past two years, such as GAN-based post filter [22], and GAN-based multi-task for TTS [23, 24].”