WaveFlow: A Compact Flow-based Model for Raw Audio

By Wei Ping, Kainan Peng, Kexin Zhao, Zhao-long Song. 2019

133 citations recorded by the source corpus.

In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
On Fast Sampling of Diffusion Probabilistic Models (2021, neutral): “In other words, one has to run the forward-pass of the neural network T times to generate a sample, which is much slower than the state-of-the-art GANs or flow-based models for image and audio synthesis (e.g., Karras et al., 2020; Kingma & Dhariwal, 2018; Kong et al., 2020a; Ping et al., 2020).”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “, 2020) and speech (Ping et al., 2020; Oord et al., 2016a) generation, distribution alignment across domains (Zhu et al.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “, 2017) and flow-based models (Prenger et al., 2019; Ping et al., 2020; Kim et al., 2019), have predominated in audio synthesis because of the simple training objective and superior ability of modeling the fine details of waveform in real data.”
A Survey on Neural Speech Synthesis (2021, methodology): “To combine the advantages of both autoregressive and bipartite transforms, WaveFlow [265] provides a unified view of likelihood-based models for audio data to explicitly trade inference parallelism for model capacity.”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “[28] have made great progress towards reducing the size of flow-based TTS models.”
It's Raw! Audio Generation with State-Space Models (2022, methodology): “(ii) Likelihood-based flow models: WaveFlow (Ping et al., 2020), WaveGlow (Prenger et al.”
End-to-End Adversarial Text-to-Speech (2020, methodology): “These can either be trained directly using maximum likelihood [30, 46, 47], or through distillation from an autoregressive model [45, 61].”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “In contrast, CARGAN (Morrison et al., 2022) incorporates the partial autoregression (Ping et al., 2020) into generator to improve the pitch and periodicity accuracy.”
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, methodology): “This new methodology, involving a generator and discriminator, has allowed for waveforms to be generated in the vocoders at a considerably higher real-time speed than in models with autoregressive constraints[16, 17], and also, waveforms can be generated with reasonable fidelity and fewer parameters”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “Compared with some autoregressive [37, 20] or flow-based [40] vocoders, it takes into account both generation quality and inference efficiency.”
LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation (2021, unknown): “[7] Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song, “WaveFlow: A compact flow-based model for raw audio,” in International Conference on Machine Learning (ICML), 2020.”
Controllable Neural Prosody Synthesis (2020, background): “Existing vocoders allow either perceptually high-quality synthesis [3, 13, 14] or a high degree of control over prosody [15].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[162] proposed WaveFlow which combines autoregressive flow and non-autoregressive convolution.”
Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, methodology): “WaveFlow (Ping et al., 2020) and Wave-Tacotron (Weiss et al.”
Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN (2021, methodology): “Another notable endeavor in this field is made by flow-based models including waveglow[12], flowavenet[13], Melflow[14], waveflow[15] and FBWAVE[16] etc.”
Improve GAN-based Neural Vocoder using Truncated Pointwise Relativistic Least Square GAN (2022, methodology): “Another notable endeavor in this field is made by flow-based models including WaveGlow[25], FloWavenet[10], Melflow[36], WaveFlow[24], WaveFit[12] and FBWAVE[30] etc.”
Non-Autoregressive Neural Text-to-Speech (2019, background): “In previous work, flow-based models have obtained stateof-the-art results for parallel waveform synthesis (van den Oord et al., 2018; Ping et al., 2018a; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2019; Ping et al., 2020).”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, background): “, 2020), speech generation (Oord et al., 2016; Ping et al., 2020), image captioning (Aneja et al.”