Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis

By Jian Cong, Shan Yang, Lei Xie, Dan Su. 2021

29 citations recorded by the source corpus.

Current two-stage TTS framework typically integrates an acoustic model with a vocoder -- the acoustic model predicts a low resolution intermediate representation such as Mel-spectrum while the vocoder generates waveform from the intermediate representation. Although the intermediate representation is served as a bridge, there still exists critical mismatch between the acoustic model and the vocoder as they are commonly separately learned and work on different distributions of representation, leading to inevitable artifacts in the synthesized speech. In this work, different from using pre-designed intermediate representation in most previous studies, we propose to use VAE combining with GAN to learn a latent representation directly from speech and then utilize a flow-based acoustic model to model the distribution of the latent representation from text. In this way, the mismatch problem is migrated as the two stages work on the same distribution. Results demonstrate that the flow-based acoustic model can exactly model the distribution of our learned speech representation and the proposed TTS framework, namely Glow-WaveGAN, can produce high fidelity speech outperforming the state-of-the-art GAN-based model.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Although ideally having advantages over two-stage systems, different end-to-end models face their own challenges and limitations: 1) no significant improvement over two-stage models in terms of voice quality, such as FastSpeech 2s [3], EATS [6], and WAVE-TACOTRON [8]; 2) slow inference speed due to ”
A Survey on Neural Speech Synthesis (2021, methodology): “, Flow, GAN, VAE, Diffusion) Besides the above acoustic models, there are a lot of other acoustic models [361, 22, 123, 183, 54], as shown in Table 2.”
Regeneration Learning: A Learning Paradigm for Data Generation (2023, unknown): “For example, in text-to-speech synthesis (Cong et al. 2021; Tan et al. 2022; Liu et al. 2022), NaturalSpeech (Tan et al. 2022) Learning Y ′ by Considering X .”
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “The acoustic vocoder is separately trained with a design similar to Kumar et al. [2024], Lee et al. [2022], Cong et al. [2021] and Liu and Qian [2021].”