Adversarial Audio Synthesis

By Chris Donahue, Julian McAuley, Miller S. Puckette. 2018

711 citations recorded by the source corpus.

Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Noise Estimation for Generative Diffusion Models (2021, methodology): “(ii) non-autoregessive models, for example, StyleGAN [15] for vision application and WaveGAN [4] for audio synthesis.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “, 2016), or inferior samples (Donahue et al., 2019) under unconditional settings.”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Our UBlock is inspired by the GAN-TTS GBlock generator, with the minor difference that we do not use BatchNorm.”
Audiobox: Unified Audio Generation with Natural Language Prompts (2023, neutral): “…and music generation: Text-to-sound (Kreuk et al., 2022) and text-to-music (Schneider et al., 2023) are the emerging paradigms for general-domain sound and music generation, in contrast to earlier studies that generate finite sound effects (Donahue et al., 2018) or instruments (Huang et al., 2018).”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “On the other hand, nonautoregressive models generate all samples independently, parallelizing the process and making it more computationally efficient (Oord et al., 2018; Prenger et al., 2019; Donahue et al., 2018).”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “Several works have attempted to unconditionally generate raw waveforms (van den Oord et al., 2016; Donahue et al., 2019) or to model the unconditional distribution of latent code or mel-spectrogram of audio (van den Oord et al.”
A Survey on Neural Speech Synthesis (2021, methodology): “GAN Generator Discriminator Loss WaveGAN [66] DCGAN [281] / WGAN-GP [94] GAN-TTS [23] / Random Window D Hinge-Loss GAN [194]”
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, background): “This problem has been addressed in the previous work (Donahue et al., 2018) by increasing receptive ﬁelds of the generator and discriminator.”
Generative Speech Coding with Predictive Variance Regularization (2021, methodology): “Although autoregressive structures for synthesis are common, feed-forward structures are used by, for example, Parallel WaveNet [3], WaveGlow [4], WaveGAN [5], and GANSynth [6].”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “For example, strided convolution models–such as SING (Defossez et al., 2018), MCNN (Arik et al., 2019), and WaveGAN (Donahue et al., 2019)–generate waveforms directly with overlapping frames.”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “We solve this problem by carefully choosing the kernel-size and stride for our deconvolutional layers as a simpler alternative to PhaseShufﬂe layer introduced in Donahue et al. (2018b).”
It's Raw! Audio Generation with State-Space Models (2022, methodology): “We collect MOS scores on digit intelligibility, audio quality and speaker diversity, as well as asking crowdworkers to classify digits following Donahue et al. [10].”
GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, background): “Early attempts [10, 11] suffered from severe upsampling artifacts.”
Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, background): “1 Note that our work is not closely related to an unsupervised waveform synthesis model, WaveGAN [13].”
High Fidelity Speech Synthesis with Adversarial Networks (2019, positive): “Two notable exceptions include WaveGAN (Donahue et al., 2019) and GANSynth (Engel et al., 2019), which both successfully applied GANs to simple datasets of audio data.”
GANSynth: Adversarial Neural Audio Synthesis (2019, methodology): “We also independently train our own waveform generating GANs off the progressive codebase and our best models achieve similar performance to WaveGAN without progressive training, so we opt to primarily show numbers from WaveGAN instead (see appendix (Engel et al., 2017), but is not a generative mode”
End-to-End Adversarial Text-to-Speech (2020, positive): “A growing body of work has applied GAN [18] variants to speech synthesis [14].”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, background): “, 2020), GAN-based models (Donahue et al., 2019; Kumar et al., 2019; Bińkowski et al., 2020; Yamamoto et al., 2020; Kong et al., 2020), and diffusion models (Kong et al.”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, background): “Non-autoregressive convo- lutional architectures can do speedy synthesis and easily capture the long-range structure in the data (Radford et al., 2015; Brock et al., 2018), but it could produce spurious high-frequency components which will hurt the audio fidelity (e.g., Donahue et al., 2018).”