WaveGrad: Estimating Gradients for Waveform Generation

By Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, William Chan. 2020

957 citations recorded by the source corpus.

This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations. Audio samples are available at this https URL.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

Classifier-Free Diffusion Guidance (2022, methodology): “We train a diffusion model in continuous time [17, 2, 8]: letting z = {zλ |λ ∈ [λmin, λmax]} for hyperparameters λmin < λmax ∈ R, the forward process q(z|x) is the variance-preserving Markov process [15] specified as”
Variational Diffusion Models (2021, neutral): “Diffusion models have recently shown impressive results in image [Ho et al., 2020, Song et al., 2021b, Nichol and Dhariwal, 2021] and audio generation [Kong et al., 2020, Chen et al., 2020] in terms of perceptual quality, but have yet to match autoregressive models on density estimation benchmarks.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b; Dhariwal and Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, methodology): “To this end, we train these super-resolution models with continuous noise conditioning, like [21, 3], and tune the noise schedule for a given number of steps during inference.”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “[8] demonstrated that diffusion models work well for audio; Jolicoeur-Martineau et al.”
Image Super-Resolution via Iterative Refinement (2021, methodology): “We also emphasize that this search is inexpensive as it does not requiremodel retraining [33].”
Noise Estimation for Generative Diffusion Models (2021, methodology): “Subsequently, in (Chen et al. 2020) a ar X iv :2 10 4.”
Symbolic Music Generation with Diffusion Models (2021, methodology): “Despite these exciting advances, DDPMs have not yet been applied to symbolic music generation because their iterative refinement sampling process is confined to continuous domains such as images [2] and audio [8,9].”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “…2017; Goyal et al., 2017; Du & Mordatch, 2019), have proven effective at generation of images (Song & Ermon, 2019; 2020; Ho et al., 2020), audio (Chen et al., 2020; Kong et al., 2020), graphs (Niu et al., 2020), and shapes (Cai ˚ Work partially done during an internship at Google Brain. et al.,…”
Denoising Diffusion Implicit Models (2020, background): “, 2020) or even a continuous time variable t (Chen et al., 2020).”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “There is a concurrent work (Chen et al., 2020) that uses diffusion probabilistic models for waveform generation.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
Improved Denoising Diffusion Probabilistic Models (2021, methodology): “…shown that this class of models can produce high-quality images (Ho et al., 2020; Song & Ermon, 2020; Jolicoeur-Martineau et al., 2020) and audio (Chen et al., 2020b; Kong et al., 2020), but it has yet to be shown that DDPMs can achieve log-likelihoods competitive with other likelihood-based…”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-image generation [7], speech synthesis [8, 9] and lossless compression [10].”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Meanwhile, diffusion models have also been widely used in speech synthesis for waveform generation [29, 30] and acoustic feature generation [31, 32].”
AudioLM: A Language Modeling Approach to Audio Generation (2022, neutral): “Recent audio synthesis models have achieved nearly veridical signal quality by leveraging methods such as autoregressive waveform modeling [1], [2], adversarial training [3]–[5] or diffusion [6], [7].”