FloWaveNet : A Generative Flow for Raw Audio

By Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, Sungroh Yoon. 2018

181 citations recorded by the source corpus.

Most modern text-to-speech architectures use a WaveNet vocoder for synthesizing high-fidelity waveform audio, but there have been limitations, such as high inference time, in its practical application due to its ancestral sampling scheme. The recently suggested Parallel WaveNet and ClariNet have achieved real-time audio synthesis capability by incorporating inverse autoregressive flow for parallel sampling. However, these approaches require a two-stage training pipeline with a well-trained teacher network and can only produce natural sound by using probability distillation along with auxiliary loss terms. We propose FloWaveNet, a flow-based generative model for raw audio synthesis. FloWaveNet requires only a single-stage training procedure and a single maximum likelihood loss, without any additional auxiliary terms, and it is inherently parallel due to the characteristics of generative flow. The model can efficiently sample raw audio in real-time, with clarity comparable to previous two-stage parallel models. The code and samples for all models, including our FloWaveNet, are publicly available.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yoshua Bengio
5. Quoc V. Le
6. Frank K. Soong
7. Junichi Yamagishi
8. Mohit Bansal

In Generative Models

1. Wei Ping
2. Jonathan Ho
3. Tim Salimans
4. Mohammad Norouzi
5. Xu Tan
6. Stefano Ermon
7. Diederik P. Kingma
8. Tao Qin

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Junichi Yamagishi
4. Tie-Yan Liu
5. Frank K. Soong
6. Wei Ping
7. Sheng Zhao
8. Tomoki Toda

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Tomoki Toda
7. Wei Ping
8. Heiga Zen

Citations in the corpus

• DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “, 2017) and flow-based models (Prenger et al., 2019; Ping et al., 2020; Kim et al., 2019), have predominated in audio synthesis because of the simple training objective and superior ability of modeling the fine details of waveform in real data.”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “…normalizing flow (NF) models like inverse autoregressive flow (IAF) (Oord et al., 2018; Ping et al., 2019), generative flow (Glow) (Prenger et al., 2019; Kim et al., 2019), and continuous normalizing flow (CNF) (Kim et al., 2020; Wu & Ling, 2020), implicit generative models like generative…”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “…of the autoregressive models, flow-based generative models (Kingma & Dhariwal, 2018) and feed-forward models have been proposed for text-to-mel-spectrogram models (Ren et al., 2019; 2021; Kim et al., 2020; Shih et al., 2021) and vocoders (Oord et al., 2018; Prenger et al., 2019; Kim et al., 2019).”
• A Survey on Neural Speech Synthesis (2021, methodology): “, Glow [163] used in WaveGlow [273], and RealNVP [64] used in FloWaveNet [159]), as shown in Table 4.”
• A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “Indeed, both WaveGlow and FloWaveNet have ≈ 100 convolutional layers, and are slow to train [35].”
• It's Raw! Audio Generation with State-Space Models (2022, methodology): “, 2019), and FloWaveNet (Kim et al., 2019) all use WaveNet as a core component of reversible flow architectures.”
• GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, background): “Particularly notable are the flow-based vocoders [5, 6] that shape a simple prior distribution into a complex data distribution conditioned on a mel spectrogram using a chain of invertible operations.”
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “[10] introduced these transformations for speech synthesis to overcome the slow sampling speed of an autoregressive vocoder, WaveNet [29].”
• High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “Another strategy is to use an invertible feed-forward neural network to model the joint density directly (Prenger et al., 2019; Kim et al., 2019).”
• Neural Spline Flows (2019, methodology): “WaveGlow [43] and FloWaveNet [25] have also successfully adapted Glow for generative modeling of audio.”
• Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “, 2019), FloWaveNet (Kim et al., 2019) and Flow++ (Ho et al.”
• End-to-End Adversarial Text-to-Speech (2020, methodology): “These can either be trained directly using maximum likelihood (Prenger et al., 2019; Kim et al., 2019; Ping et al., 2019b), or through distillation from an autoregressive model (van den Oord et al.”
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “…et al., 2016; Mehri et al., 2017; Kalchbrenner et al., 2018), flow-based models (van den Oord et al., 2018; Ping et al., 2019; Prenger et al., 2019; Kim et al., 2019; Ping et al., 2020; Lee et al., 2020), GAN-based models (Donahue et al., 2019; Kumar et al., 2019; Bi´nkowski et al., 2020;…”
• WaveFlow: A Compact Flow-based Model for Raw Audio (2019, methodology): “, 2019) and FloWaveNet (Kim et al., 2019) have a simple training pipeline as they solely use the maximum likelihood objective.”
• Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “The other family is flow-based models [10, 11, 12], including WaveGlow [13] and FloWaveNet [14].”
• Quasi-Periodic Parallel WaveGAN Vocoder: A Non-autoregressive Pitch-dependent Dilated Convolution Model for Parametric Speech Generation (2020, methodology): “Unlike flow-based models [15–19] adopting an invertible network to transform the distribution of the input noise to the real data distribution, PWG adopts a GAN structure to make the generator learn the transformation via the feedback from the discriminator.”
• Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework (2019, methodology): “, 2016) or non-autoregressive (Oord et al., 2018; Ping et al., 2018; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2020).”