GANSynth: Adversarial Neural Audio Synthesis

By Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, Adam Roberts. 2019

425 citations recorded by the source corpus.

Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure at the expense of global latent structure and slow iterative sampling, while Generative Adversarial Networks (GANs), have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

Subfields: Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “…(Hiller Jr & Isaacson, 1957; Moorer, 1972; Hadjeres et al., 2017; Huang et al., 2017), digital vocoders generating a singer’s voice (Bonada & Serra, 2007; Saino et al., 2006; Blaauw & Bonada, 2017) and also synthesizers producing timbres for various musical instruments (Engel et al., 2017; 2019).”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “…for training, such as ﬂow-based models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng et al., 2020), and generative adversarial network (GAN) based models (Kumar et al., 2019; Bi´nkowski et al., 2020; Yamamoto et al., 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Our UBlock is inspired by the GAN-TTS GBlock generator, with the minor difference that we do not use BatchNorm.”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “Also, the initial studies using GANs to generate invertible spectrograms involved estimating instantaneous frequency (Engel et al., 2018).”
Generative Speech Coding with Predictive Variance Regularization (2021, methodology): “Although autoregressive structures for synthesis are common, feed-forward structures are used by, for example, Parallel WaveNet [3], WaveGlow [4], WaveGAN [5], and GANSynth [6].”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “Model Size: Table B.6, compares parameter counts for the DDSP models and comparable models including GANSynth (Engel et al., 2019), WaveRNN (Hantrakul et al., 2019), and a WaveNet Autoencoder (Engel et al., 2017).”
Adversarial Audio Synthesis (2018, methodology): “For a more thorough investigation of spectrogram generation methods, we point to follow-up work (Engel et al., 2019).”
High Fidelity Speech Synthesis with Adversarial Networks (2019, positive): “2019) and GANSynth (Engel et al., 2019), which both successfully applied GANs to simple datasets of audio data.”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, background): “Recent works have used generative adversarial networks (GANs) [14] to model both waveforms and spectral representations [12, 13].”
WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN (2019, background): “GANSynth [14] has also been proposed to synthesize high quality musical audio using GANs.”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, background): “, 2018), audio (Engel et al., 2019; 2017), and music (Roberts et al.”
ItôWave: Itô Stochastic Differential Equation is all You Need for Wave Generation (2022, methodology): “mainly focus on developing non-AR wave generation models, such as Parallel WaveNet [4], GanSynth [5], MelGan [6], WaveGlow [7], Parallel WaveGan [8], and so on.”
Artificial intelligence in the creative industries: a review (2020, neutral): “Alongside these methods of musical notation based audio synthesis, there also exists a range of direct waveform synthesis techniques that learn and/or act directly on the waveform of the audio itself (for example [31, 32].”
Musika! Fast Infinite Waveform Music Generation (2022, positive): “Systems such as WaveGAN and Spec-GAN [12], GANSynth [13], DrumGAN [14] and MP3Net [15] attempt to generate audio of a ﬁxed length using various architectures of Generative Adversarial Networks [16] (GANs).”
Conditional Spoken Digit Generation with StyleGAN (2020, background): “Convolutional GAN has demonstrated a capability to synthesize various raw audio sounds (WaveGAN) [6], although at limited quality, while [7] applied GANs on pitchconditioned instrument sound synthesis.”