High Fidelity Speech Synthesis with Adversarial Networks

By Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, Karen Simonyan. 2019

264 citations recorded by the source corpus.

Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Frechet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at this https URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Noise Estimation for Generative Diffusion Models (2021, methodology): “Subsequently, in [2] a generative neural diffusion process based on score matching was applied to speech generation, obtaining state of the art results in comparison to well-established methods, such as Wavenet [23], Wavernn [14] and GANTTS [1].”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “…for training, such as ﬂow-based models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng et al., 2020), and generative adversarial network (GAN) based models (Kumar et al., 2019; Bi´nkowski et al., 2020; Yamamoto et al., 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “The upsampling block (UBlock) of WaveGrad follows the GAN-TTS generator, with a minor difference that no BatchNorm is used.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “While WaveNet inspired more computationally efficient alternatives such as WaveRNN [2] or parallel WaveNet [20], a significant paradigm shift occurred with the introduction of adversarial audio generation [3], [4], [21], which enables high fidelity generation without any autoregressive component.”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, positive): “For efficient end-to-end vocoder training, we do not use the entire latent variable z as input to the vocoder, instead, we randomly sample constant length partial sequences from z [11, 25, 26, 18].”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “…(Ho et al., 2020; Chen et al., 2021a; Kong et al., 2021; Popov et al., 2021; Jeong et al., 2021), and GAN based models (Goodfellow et al., 2014; Kumar et al., 2019; Bińkowski et al., 2019; Kong et al., 2020) have been proposed as high-quality speech synthesis models with parallel sampling schemes.”
A Survey on Neural Speech Synthesis (2021, methodology): “GAN Generator Discriminator Loss WaveGAN [66] DCGAN [281] / WGAN-GP [94] GAN-TTS [23] / Random Window D Hinge-Loss GAN [194]”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “GAN-based feed-forward networks with multiple discriminators, each distinguishing samples at different scales or periods, achieve high-quality raw waveform synthesis (Kumar et al., 2019; Bińkowski et al., 2019; Kong et al., 2020).”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “• As in (Bińkowski et al., 2020; Donahue et al., 2021), we use random mel-spectrogram segments of fixed length (2 seconds in our case) as training targets y to allow for memory-efficient training.”
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, methodology): “For previous work using multi-discriminator architecture such as MPD and MSD, Bińkowski et al. (2019)’s work can also be referred.”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, neutral): “[3], we further propose a new model for generating audio using an efficient overlap-add upsampling module.”
It's Raw! Audio Generation with State-Space Models (2022, background): “, 2020) and GANTTS (Bińkowski et al., 2020), which use WaveNets in their discriminators.”
End-to-End Adversarial Text-to-Speech (2020, methodology): “075 GAN-TTS (Bińkowski et al., 2020) 4.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “In contrast, GANTTS [3] uses an ensemble of discriminators which operate on random windows of different sizes, and enforces the mapping between the conditioner and waveform adversarially using conditional discriminators.”
Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “The input of MelGAN and Parallel WaveGAN is mel-spectrogram, while the input of GAN-TTS is linguistic features.”
Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, background): “Although VAEs and GANs (Hsu et al., 2018; Bi ´ nkowski et al., 2019; Akuzawa et al., 2018) based models also provide a latent prior that can be easily manipulated, in Flowtron this comes at no cost in speech quality nor optimization challenges.”
Fastpitch: Parallel Text-to-Speech with Pitch Prediction (2020, methodology): “In addition to Transformer models investigated in this work [21], convolutional GAN-TTS [2] is able to synthesize raw audio waveforms with state-of-the-art quality.”