VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network

By Jinhyeok Yang, Young-Ik Kim, Hoonyoung Cho, Injung Kim. 2020

78 citations recorded by the source corpus.

We present a novel high-fidelity real-time neural vocoder called VocGAN. A recently developed GAN-based vocoder, MelGAN, produces speech waveforms in real-time. However, it often produces a waveform that is insufficient in quality or inconsistent with acoustic characteristics of the input mel spectrogram. VocGAN is nearly as fast as MelGAN, but it significantly improves the quality and consistency of the output waveform. VocGAN applies a multi-scale waveform generator and a hierarchically-nested discriminator to learn multiple levels of acoustic properties in a balanced way. It also applies the joint conditional and unconditional objective, which has shown successful results in high-resolution image synthesis. In experiments, VocGAN synthesizes speech waveforms 416.7x faster on a GTX 1080Ti GPU and 3.24x faster on a CPU than real-time. Compared with MelGAN, it also exhibits significantly improved quality in multiple evaluation metrics including mean opinion score (MOS) with minimal additional overhead. Additionally, compared with Parallel WaveGAN, another recently developed high-fidelity vocoder, VocGAN is 6.98x faster on a CPU and exhibits higher MOS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Frank K. Soong
7. Yu Zhang
8. Jinglin Liu

In Computer Vision and Pattern Recognition

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Luc Van Gool
5. Mohammad Norouzi
6. Dacheng Tao
7. Xiaogang Wang
8. Ming-Hsuan Yang

In Statistical and Nonlinear Physics

1. Mohammad Norouzi
2. William Chan
3. Tao Qin
4. Dong Yu
5. Dan Su
6. Yi Ren
7. Heiga Zen
8. Frank K. Soong

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Tao Qin
4. Zhou Zhao
5. Tie-Yan Liu
6. Jinglin Liu
7. Rongjie Huang
8. Frank K. Soong

Citations in the corpus

• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…generative adversarial networks (GAN) (Donahue et al., 2018; Engel et al., 2019; Kumar et al., 2019; Yamamoto et al., 2020; Bińkowski et al., 2020; Yang et al., 2020a;b; McCarthy & Ahmed, 2020) and energy score (Gritsenko et al., 2020), variational auto-encoder models (Peng et al., 2020), as…”
• A Survey on Neural Speech Synthesis (2021, methodology): “Specifically, VocGAN [401] proposes a multi-scale generator that can gradually output waveform sequence at different scales, from coarse-grained to fine-grained.”
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, methodology): “However, recent GAN vocoders made breakthrough improvements [13, 14, 15, 16, 17], often exceeding the previous state-of-the-art WaveNet on various metrics both qualitatively (e.”
• Improved Parallel Wavegan Vocoder with Perceptually Weighted Spectrogram Loss (2021, methodology): “Although it is possible to train GAN-based non-autoregressive models by only using adversarial loss function [12], employing the MRSTFT loss function has been proven to be advantageous for increasing the training efficiency [10, 13, 14].”
• DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs (2022, methodology): “The discriminator, denoted as D φ ( x t − 1 , x t , t, s ) with learnable parameters φ , is modeled as joint conditional and unconditional (JCU)(Yang et al., 2020; 2021).”
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN (2021, methodology): “GAN-based methods([17],[18],[19],[20],[21],[22],[23]) are promising given some models can even synthesize waves in realtime on a single GPU and achieve a higher MOS at the same time, suited for actual industrial deployment.”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, background): “, VocGAN (Yang et al. 2020b), Style MelGAN (Mustafa, Pia, and Fuchs 2020), Universal MelGAN (Jang, Lim, and Yoon 2020)) have been proposed to further improve voice quality.”
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus (2021, unknown): “Previous work like VocGAN [41] has demonstrated the efficiency of JCU loss for adversarial training.”
• Avocodo: Generative Adversarial Network for Artifact-free Vocoder (2022, methodology): “…on generative adversarial network (GAN) (Goodfellow et al. 2014) with non-autoregressive convolutional architectures have been proposed (Yamamoto, Song, and Kim 2020; Kumar et al. 2019; Yang et al. 2020; Kong, Kim, and Bae 2020; Yang et al. 2021; Mustafa, Pia, and Fuchs 2021; Kim et al. 2021).”
• Noise Level Limited Sub-Modeling for Diffusion Probabilistic Vocoders (2021, methodology): “The other type comprises generative adversarial network (GAN)-based models [15] such as MelGAN [16], Parallel WaveGAN (PWG) [17], GAN-TTS [18,19], VocGAN [20], HiFi-GAN [21], and Multi-band MelGAN [22].”
• ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “A GAN [9]-based model [10, 11, 12, 23, 24, 25] achieves parallelizable training and inference through noncausal convolutions.”