Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech

By Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, Lei Xie. 2020

238 citations recorded by the source corpus.

In this paper, we propose multi-band MelGAN, a much faster waveform generation model targeting to high-quality text-to-speech. Specifically, we improve the original MelGAN by the following aspects. First, we increase the receptive field of the generator, which is proven to be beneficial to speech generation. Second, we substitute the feature matching loss with the multi-resolution STFT loss to better measure the difference between fake and real speech. Together with pre-training, this improvement leads to both better quality and better training stability. More importantly, we extend MelGAN with multi-band processing: the generator takes mel-spectrograms as input and produces sub-band signals which are subsequently summed back to full-band signals as discriminator input. The proposed multi-band MelGAN has achieved high MOS of 4.34 and 4.22 in waveform generation and TTS, respectively. With only 1.91M parameters, our model effectively reduces the total computational complexity of the original MelGAN from 5.85 to 0.95 GFLOPS. Our Pytorch implementation can achieve a real-time factor of 0.03 on CPU without hardware specific optimization.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Yu Zhang
7. Sheng Zhao
8. Junichi Yamagishi

In Generative Models

1. Wei Ping
2. Xu Tan
3. Tao Qin
4. Mohammad Norouzi
5. Tie-Yan Liu
6. Jonathan Ho
7. William Chan
8. Frank K. Soong

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Junichi Yamagishi
7. Wei Ping
8. Heiga Zen

Citations in the corpus

• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “…like generative adversarial network (GAN) (Donahue et al., 2018; Engel et al., 2019; Kumar et al., 2019; Yamamoto et al., 2020; Bikowski et al., 2020; Yang et al., 2020; McCarthy & Ahmed, 2020) and energy score (Gritsenko et al., 2020), variational models like variational auto-encoder (VAE)…”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, background): “Multi-band processing was earlier proposed in [43] to predict audio in sub-bands which are subsequently summed to produce the full-band audio.”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “As for the vocoders, generative adversarial network (GAN) [6] based vocoders, such as multi-band Mel-GAN [7] and HifiGAN [8], are widely used for their high quality of speech and fast generation speed.”
• A Survey on Neural Speech Synthesis (2021, methodology): “WaveGAN [396] Voc melS FF−→wav ICASSP20 2019.10 DCA-Tacotron [17] AM ph AR−→melS AR−→wav ICASSP20 2019.10 WaveFlow [265] Voc melS AR−→wav ICML20 2019.12 SqueezeWave [426] Voc melS FF−→wav arXiv20 2020.01 AlignTTS [422] AM ch/ph FF−→melS FF−→wav ICASSP20 2020.03 RobuTrans [190] AM ph AR−→melS AR−→wav”
• RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, methodology): “Since we target the modelling of 48kHz audio signals, we leverage a multiband decomposition of the raw waveform (see appendix B) as a way to decrease the temporal dimensionality of the data (Yang et al., 2020).”
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, methodology): “For missing details, we refer to MelGAN and MultiBand MelGAN [21], which are the baseline models of UMGAN.”
• End-to-End Adversarial Text-to-Speech (2020, positive): “Spectrogram prediction losses have been used extensively for feed-forward audio prediction models [3, 16, 64, 67, 68, 69].”
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “Yang et al. (2021) and Mustafa et al. (2021) further improve MelGAN by incorporating the multi-resolution STFT loss.”
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, positive): “We prepared the following GAN-based vocoders for comparison: MelGAN, Parallel WaveGAN, and HiFi-GAN.”
• ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “A GAN [9]-based model [10, 11, 12, 23, 24, 25] achieves parallelizable training and inference through noncausal convolutions.”
• HUI-Audio-Corpus-German: A high quality TTS dataset (2021, methodology): “Performance of text to speech (TTS) systems has increased vastly over the past decade, primarily by leveraging deep neural networks (DNNs) [1], which in turn lead to higher acceptance by end-users [1, 2].”
• A Streamwise Gan Vocoder for Wideband Speech Coding at Very Low Bit Rate (2021, unknown): “SSMGAN further reduces the complexity compared to the baseline model by introducing multiband synthesis as in [19, 20].”
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN (2021, methodology): “As suggested in [19], we replace feature matching withmulti-resolution STFT loss.”
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus (2021, unknown): “2) Several parallel generation meth-ods [40, 42] have been proposed to speed up waveforms synthesis.”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, unknown): “After division by the differentiable Pseudo Quadrature Mirror Filter (PQMF) bank [34], the four sub-band signals are respectively fed into the multiple local-level discriminators.”
• An Efficient Subband Linear Prediction for LPCNet-Based Neural Synthesis (2020, background): “Some related works have investigated the neural vocoders in the subband domain, such as subband WaveNet[16, 17, 18], subband FFTNet[19], subband WaveRNN[20], FeatherWave[21], and multiband MelGan[22].”