DiffWave: A Versatile Diffusion Model for Audio Synthesis

By Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro. 2020

2,005 citations recorded by the source corpus.

In this work, we propose DiffWave, a versatile Diffusion probabilistic model for conditional and unconditional Waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in Different Waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality~(MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Signal Processing

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “results in image synthesis [29,82] and beyond [7,44,47,56], and define the state-of-the-art in class-conditional image synthesis [15,30] and super-resolution [70].”
Classifier-Free Diffusion Guidance (2022, methodology): “These models have delivered audio synthesis performance rivaling the quality of autoregressive models with substantially fewer inference steps [2, 9], and they have delivered ImageNet generation results outperforming BigGAN-deep [1] and VQ-VAE-2 [11] in terms of FID score and classification accuracy”
Variational Diffusion Models (2021, neutral): “Diffusion models have recently shown impressive results in image [Ho et al., 2020, Song et al., 2021b, Nichol and Dhariwal, 2021] and audio generation [Kong et al., 2020, Chen et al., 2020] in terms of perceptual quality, but have yet to match autoregressive models on density estimation benchmarks.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b; Dhariwal and Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “Diffusion models [24] have recently been shown to synthesize high quality images and audio [3, 10, 14, 27], which have long been led by other classes of generative models such as autoregressive models, GANs, VAEs, and flows [2, 5, 7, 9, 12, 13, 18, 29, 30, 32].”
Noise Estimation for Generative Diffusion Models (2021, background): “A parallel contribution presented high fidelity speech generation results using a different neural diffusion process [17].”
Symbolic Music Generation with Diffusion Models (2021, background): “, 2020) are a promising new class of generative models that can generate comparably high-quality samples (Chen et al., 2020; Kong et al., 2020) by learning to invert a diffusion process from data to Gaussian noise.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “…Goyal et al., 2017; Du & Mordatch, 2019), have proven effective at generation of images (Song & Ermon, 2019; 2020; Ho et al., 2020), audio (Chen et al., 2020; Kong et al., 2020), graphs (Niu et al., 2020), and shapes (Cai ˚ Work partially done during an internship at Google Brain. et al., 2020).”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “With these important improvements, DPMs show great potential in various applications, including speech synthesis (Chen et al., 2020; Kong et al., 2020; Popov et al., 2021; Lam et al., 2021), controllable generation (Choi et al.”
Improved Denoising Diffusion Probabilistic Models (2021, background): “, 2020) and audio (Chen et al., 2020b; Kong et al., 2020), but it has yet to be shown that DDPMs can achieve loglikelihoods competitive with other likelihood-based models such as autoregressive models (van den Oord et al.”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Diffusion-based generative models Recent advances in diffusion models [59] have enabled state-of-the-art image synthesis [10, 18, 19, 53, 55, 60] as well as generative models of other modalities such as video [21, 58], audio [30], text [35] and network parameters [45].”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “They are also rapidly finding use in other domains such as audio [27, 37] and video [19] generation, image segmentation [4, 54] and language translation [34].”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing ﬂow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Meanwhile, diffusion models have also been widely used in speech synthesis for waveform generation [29, 30] and acoustic feature generation [31, 32].”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Speech/Audio Synthesis With the rapid development and application of neural networks, speech and audio synthesis have made tremendous progress with different network frameworks, such as WaveNet [Oord et al., 2016], HiFi-GAN [Kong et al., 2020a], and Diffwave Kong et al. [2020b].”
AudioLM: A Language Modeling Approach to Audio Generation (2022, neutral): “Recent audio synthesis models have achieved nearly veridical signal quality by leveraging methods such as autoregressive waveform modeling [1], [2], adversarial training [3]–[5] or diffusion [6], [7].”