WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

By Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, William Chan. 2021

102 citations recorded by the source corpus.

This paper introduces WaveGrad 2, a non-autoregressive generative model for text-to-speech synthesis. WaveGrad 2 is trained to estimate the gradient of the log conditional density of the waveform given a phoneme sequence. The model takes an input phoneme sequence, and through an iterative refinement process, generates an audio waveform. This contrasts to the original WaveGrad vocoder which conditions on mel-spectrogram features, generated by a separate model. The iterative refinement process starts from Gaussian noise, and through a series of refinement steps (e.g., 50 steps), progressively recovers the audio sequence. WaveGrad 2 offers a natural way to trade-off between inference speed and sample quality, through adjusting the number of refinement steps. Experiments show that the model can generate high fidelity audio, approaching the performance of a state-of-the-art neural TTS system. We also report various ablation studies over different model configurations. Audio samples are available at https://wavegrad.github.io/v2.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-image generation [7], speech synthesis [8, 9] and lossless compression [10].”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Correspondence to: Sungroh Yoon <sryoon@snu.ac.kr>. raw waveforms from text at once (Kim et al., 2021; Weiss et al., 2021; Chen et al., 2021b).”
Palette: Image-to-Image Diffusion Models (2021, methodology): “Conditional diffusion models [Chen et al. 2021a; Saharia et al. 2021] make the denoising process conditional on an input signal.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, background): “…2022, Ding et al., 2021], image-to-image generation [Saharia et al., 2022c,a, Whang et al., 2022], 3D synthesis [Poole et al., 2022, Watson et al., 2022], text-to-speech [Chen et al., 2021a, Kong et al., 2021, Chen et al., 2021b], and density estimation [Kingma et al., 2021, Song et al., 2021a].”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[89] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “In contrast to WaveGrad [7] converting Mel-spectrogram to waveform, WaveGrad 2 [8] adopts an end-to-end manner that takes a phoneme sequence as input and generates the audio directly.”
Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives (2021, background): “…2020; Vahdat et al., 2021), image superresoution (Saharia et al., 2021; Ho et al., 2021), image translation (Sasaki et al., 2021), speech synthesis (Chen et al., 2020, 2021; Kong et al., 2021; Popov et al., 2021), symbolic music generation (Mittal et al., 2021), natural language generation…”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, unknown): “…et al., 2022a; Dhariwal & Nichol, 2022) including density estimation (Kingma et al., 2021), text-to-speech (Chen et al., 2021a; Kong et al., 2021; Chen et al., 2021b), image-to-image (Saharia et al., 2022c;a; Whang et al., 2022), and text-to-image (Rombach et al., 2022; Nichol et al., 2021;…”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, background): “In principle, this should only be possible for DDPMs trained with continuous time sampling (Chen et al., 2021a; Song et al., 2021b; Kingma et al., 2021), but in practice we find that DDPMs trained with continuously embedded discrete timesteps are still well-behaved when applied at timesteps not…”
Deblurring via Stochastic Refinement (2021, methodology): “For conditional models that must estimate p ( x | y ) , we make f θ accept y as the conditioning input, as was done in [13, 58].”