Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram

By Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim. 2019

1,014 citations recorded by the source corpus.

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Wei-Ning Hsu
5. Frank K. Soong
6. Dong Yu
7. Jinyu Li
8. Junichi Yamagishi

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Wei-Ning Hsu
4. Sheng Zhao
5. Yossi Adi
6. Lei He
7. Yi Ren
8. Tie-Yan Liu

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Yossi Adi
7. Yi Ren
8. Zhou Zhao

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “Parallel WaveGAN (Yamamoto et al., 2020) and MelGAN (Kumar et al.”
• DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “…for training, such as flow-based models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng et al., 2020), and generative adversarial network (GAN) based models (Kumar et al., 2019; Bi´nkowski et al., 2020; Yamamoto et al., 2020).”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Our UBlock is inspired by the GAN-TTS GBlock generator, with the minor difference that we do not use BatchNorm.”
• High Fidelity Neural Audio Compression (2022, background): “…et al., 2018; Goel et al., 2022), the most relevant ones here are those based on Generative Adversarial Networks (GAN) (Kumar et al., 2019; Yamamoto et al., 2020a; Kong et al., 2020; Andreev et al., 2022) were able to match the quality of autoregressive by combining various adversarial…”
• FedSpeech: Federated Text-to-Speech with Continual Learning (2021, methodology): “Aiming to synthesize natural speech, text to speech (TTS) [Shen et al. , 2018; Yamamoto et al. , 2020] remains a crucial research topic.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “The reconstruction performance is determined jointly by the encoding [65] that is used by A3T [3].”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, background): “Frequency domain reconstruction loss: while the mel-reconstruction loss [18] is known to improve stability, fidelity and convergence speed, the multi-scale spectral losses[42, 11, 15] encourage modeling of frequencies in multiple time-scales.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, background): “Codec Decoder Loss Codec decoder(vocoder) in end-to-end training has a multi-resolution spectrogram loss Lmrs, adversarial loss LAdv and feature match loss Lfm [21, 17], which helps to generate realistic results when jointly optimizing with adversarial loss functions.”
• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “However, due to the absence of speech contextual information, the boundaries of the modified region would be not (2)We focus on the text-to-spectrogram stage and use an off-theshelf vocoder Parallel WaveGAN (Yamamoto et al., 2020).”
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “For the waveform synthesis, we use the Parallel WaveGAN (Yamamoto et al., 2020), a non-autoregressive variant of the WaveNet vocoder.”
• A Survey on Neural Speech Synthesis (2021, methodology): “These additional losses can improve the stability and efficiency of adversarial training [396], and improve the perceptual audio quality.”
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, background): “Eventually, parallel GAN-based vocoders such as Parallel Wave-GAN (Yamamoto et al., 2020), MelGAN (Kumar et al., 2019), and HiFi-GAN (Kong et al., 2020) greatly improved the performance of waveform generation on CPU devices.”
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, methodology): “itecture for discriminators operating on multiple scales of raw waveforms. With sophisticated architectural consideration, the MelGAN generator is compact enough to enable real-time synthesis on CPU. Yamamoto et al. (2020) proposed multi-resolution STFT loss function to improve and stabilize GAN tra”
• A Spectral Energy Distance for Parallel Speech Synthesis (2020, neutral): “[38] for speech synthesis, and by Engel et al.”
• It's Raw! Audio Generation with State-Space Models (2022, background): “(iv) Generative adversarial networks (GAN): Parallel WaveGAN (Yamamoto et al., 2020) and GANTTS (Bińkowski et al.”
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, background): “However, recent GAN vocoders made breakthrough improvements [13, 14, 15, 16, 17], often exceeding the previous state-of-the-art WaveNet on various metrics both qualitatively (e.”
• Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, background): “Similarly to [24, 25], we define the STFT loss to be the sum of the spectral convergence (sc) loss and the magnitude loss as follows,”