Waveglow: A Flow-based Generative Network for Speech Synthesis

By Ryan J. Prenger, Rafael Valle, Bryan Catanzaro. 2018

1,161 citations recorded by the source corpus.

In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow [1] and WaveNet [2] in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a single network, trained using only a single cost function: maximizing the likelihood of the training data, which makes the training procedure simple and stable. Our PyTorch implementation produces audio samples at a rate of more than 500 kHz on an NVIDIA V100 GPU. Mean Opinion Scores show that it delivers audio quality as good as the best publicly available WaveNet implementation. All code will be made publicly available online [3].

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that ha”
Jukebox: A Generative Model for Music (2020, background): “WaveGlow (Prenger et al., 2019) is a ﬂow-based model for parallel sample-level audio synthesis, which can be trained with a straightforward maximum-likelihood estimation and thus is advantageous to the two-stage training process needed for distillation.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…2020; Chen et al., 2020; Ramesh et al., 2021), audio wave-forms conditioned on language features (van den Oord et al., 2016a; Oord et al., 2017; Prenger et al., 2019; Bi´nkowski et al., 2019), natural language in the form of text (Rad-ford et al., 2019; Brown et al., 2020), and music…”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “, 2017) and flow-based models (Prenger et al., 2019; Ping et al., 2020; Kim et al., 2019), have predominated in audio synthesis because of the simple training objective and superior ability of modeling the fine details of waveform in real data.”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “…generation, including normalizing ﬂow (NF) models like inverse autoregressive ﬂow (IAF) (Oord et al., 2018; Ping et al., 2019), generative ﬂow (Glow) (Prenger et al., 2019; Kim et al., 2019), and continuous normalizing ﬂow (CNF) (Kim et al., 2020; Wu & Ling, 2020), implicit generative models like…”
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “, 2018] model and the WaveGlow [Prenger et al., 2019] vocoder.”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Later, end-to-end neural TTS models were introduced, which synthe-size Mel spectrograms and employ a vocoder (Oord et al., 2017; Prenger et al., 2019) for speech synthesis (Wang et al., 2017; Arık et al., 2017; Ren et al., 2019).”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “To mitigate this problem, audio generation is typically divided into two stages: 1) predicting audio conditioned on some intermediate representation such as mel-spectrograms [23, 27, 19, 29] and 2) predicting the intermediate representation given some conditioning information, such as text [34, 33].”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “On the other hand, nonautoregressive models generate all samples independently, parallelizing the process and making it more computationally efficient (Oord et al., 2018; Prenger et al., 2019; Donahue et al., 2018).”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “…(linear-or mel-spectrogram) spectrogram-to-wave stage (vocoder) aforementioned acoustic representation into actual wave signals (Oord et al., 2018; Prenger et al., 2019). and unseen-speaker settings, the existing to be trained with an additional embedding (Jia et al., 2018), which is extracted…”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, positive): “The Posterior Encoder consists of 16 noncausal WaveNet residual blocks (Prenger et al., 2019; Kim et al., 2020).”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “…of the autoregressive models, flow-based generative models (Kingma & Dhariwal, 2018) and feed-forward models have been proposed for text-to-mel-spectrogram models (Ren et al., 2019; 2021; Kim et al., 2020; Shih et al., 2021) and vocoders (Oord et al., 2018; Prenger et al., 2019; Kim et al., 2019).”
A Survey on Neural Speech Synthesis (2021, methodology): “, Glow [163] used in WaveGlow [273], and RealNVP [64] used in FloWaveNet [159]), as shown in Table 4.”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “For the posterior encoder, we use the non-causal WaveNet residual blocks used in WaveGlow (Prenger et al., 2019) and Glow-TTS (Kim et al.”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “Popular architectures based on Normalizing Flows like Parallel WaveNet (van den Oord et al., 2018) and Wave-Glow (Prenger et al., 2019) managed to accelerate inference while keeping synthesis quality at a very high level but demonstrated fast synthesis on GPU devices only.”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, positive): “Next, a vocoder is applied to reconstruct the phase from the Mel-spectrogram (e.g., Griffin-Lim [6], WaveNet [7], WaveGlow [8], or HiFi-GAN [9]).”
On Generative Spoken Language Modeling from Raw Audio (2021, positive): “For waveform generation, we use the pre-trained ﬂow-based neural vocoder WaveG-low (Prenger et al., 2019).”
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, methodology): “WaveGlow (Prenger et al., 2019) eliminates the need for distilling a teacher model, and simpliﬁes the learning process through maximum likelihood estimation by employing efﬁcient bijective ﬂows based on Glow (Kingma and Dhariwal, 2018).”
Generative Speech Coding with Predictive Variance Regularization (2021, methodology): “Although autoregressive structures for synthesis are common, feed-forward structures are used by, for example, Parallel WaveNet [3], WaveGlow [4], WaveGAN [5], and GANSynth [6].”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, neutral): “Indeed, both WaveGlow and FloWaveNet have ≈ 100 convolutional layers, and are slow to train [35].”