MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

By Kundan Kumar, Rithesh Kumar, T. Boissiere, L. Gestin, Wei Zhen Teoh, Jose M. R. Sotelo, Alexandre de Brébisson, Yoshua Bengio, Aaron C. Courville. 2019

1,145 citations recorded by the source corpus.

Previous works (Donahue et al., 2018a; Engel et al., 2019a) have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “Parallel WaveGAN (Ya-mamoto et al., 2020) and MelGAN (Kumar et al., 2019) are GAN-based approaches directly modeling audio wave-forms, achieving similar quality as WaveNet and WaveGlow models with signiﬁcantly fewer parameters.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “…for training, such as ﬂow-based models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng et al., 2020), and generative adversarial network (GAN) based models (Kumar et al., 2019; Bi´nkowski et al., 2020; Yamamoto et al., 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Our UBlock is inspired by the GAN-TTS GBlock generator, with the minor difference that we do not use BatchNorm.”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “For the wave-based discriminator, we use the same multi-resolution convolutional discriminator proposed in [15] and adopted in [45].”
High Fidelity Neural Audio Compression (2022, methodology): “…et al., 2020a; Kalchbrenner et al., 2018; Goel et al., 2022), the most relevant ones here are those based on Generative Adversarial Networks (GAN) (Kumar et al., 2019; Yamamoto et al., 2020a; Kong et al., 2020; Andreev et al., 2022) were able to match the quality of autoregressive by combining…”
GAN Memory with No Forgetting (2020, background): “Generative adversarial networks (GANs) GANs have shown increasing power to synthesize highly realistic observations [32, 45, 51, 9, 33, 34], and have found wide applicability in various fields [39, 1, 19, 81, 83, 84, 12, 89, 37].”
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “This is followed by a residual vector quantizer (RVQ) [3, 4] to obtain parallel token streams.”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “Generative modeling of high-resolution audio is difficult due to high dimensionality (~44,100 samples per second of audio) [23, 19], and presence of structure at different time-scales with both short and long-term dependencies.”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “Progress was made with the introduction of advanced critics, such as the multi-scale discriminator (MSD) (Kumar et al., 2019) and the multi-period discriminator (MPD) (Kong et al., 2020).”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “While WaveNet inspired more computationally efficient alternatives such as WaveRNN [2] or parallel WaveNet [20], a significant paradigm shift occurred with the introduction of adversarial audio generation [3], [4], [21], which enables high fidelity generation without any autoregressive component.”
DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Codec Decoder Loss Codec decoder(vocoder) in end-to-end training has a multi-resolution spectrogram loss Lmrs, adversarial loss LAdv and feature match loss Lfm [21, 17], which helps to generate realistic results when jointly optimizing with adversarial loss functions.”
Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “We use MelGAN [35] as our neural vocoder during inference, which is pre-trained on an internal 6-speaker dataset with 3 male and 3 female speakers by the MelGAN’s authors.”
A Survey on Neural Speech Synthesis (2021, methodology): “WaveNet [249], DeepVoice 3 [264], ClariNet [263], ParaNet [262], MelGAN [174], EATS [67], DCTTS [326], WaveGlow [273], FloWaveNet [159]”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “GAN-based feed-forward networks with multiple discriminators, each distinguishing samples at different scales or periods, achieve high-quality raw waveform synthesis (Kumar et al., 2019; Bi ´ nkowski et al., 2019; Kong et al., 2020).”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, background): “, 2020), MelGAN (Kumar et al., 2019), and HiFi-GAN (Kong et al.”
AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “We compare AdaSpeech with several settings: 1) GT, the ground-truth recordings; 2) GT mel + Vocoder, using ground-truth mel-spectrogram to synthesize waveform with MelGAN vocoder; 3) Baseline (spk emb), a baseline system based on FastSpeech2 which only ﬁne-tunes the speaker embedding during adaptati”