Parallel WaveNet: Fast High-Fidelity Speech Synthesis

By Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg and 12 more. 2017

912 citations recorded by the source corpus.

The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system. However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting. This paper introduces Probability Density Distillation, a new method for training a parallel feed-forward network from a trained WaveNet with no significant difference in quality. The resulting system is capable of generating high-fidelity speech samples at more than 20 times faster than real-time, and is deployed online by Google Assistant, including serving multiple English and Japanese voices.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

End-to-End Object Detection with Transformers (2020, methodology): “Compared to most previous work on direct set prediction, the main features of DETR are the conjunction of the bipartite matching loss and transformers with (non-autoregressive) parallel decoding [29,12,10,8].”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “, 2016) — a state-of-the-art audio generative model used to serve production speech synthesis applications at Google (Oord et al., 2018).”
Jukebox: A Generative Model for Music (2020, methodology): “To do so, we use a deep residual WaveNet (Xie et al., 2017) followed by an upsampling strided convolution and a layer norm (Ba et al., 2016), and add the output as extra positional information to the embed-dings of the current level.”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Another approach approximates an autoregressive density model via distillation (van den Oord et al., 2018), but this method provides no guarantees that samples from the distilled model originate from the original model distribution.”
Stochastic Normalizing Flows (2020, background): “Moreover, having access to tractable density is useful in variational inference (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Louizos & Welling, 2017; Berg et al., 2018) or approximate sampling from distributions given by an energy function (Oord et al., 2017).”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “, 2016b) have (a) and possibly (c) (Oord et al., 2017) but can’t be trained efficiently under maximum-likelihood framework; flow models enjoy efficient generation (a) and inference (b) but are lacking behind in terms of capacity.”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “This WN architecture is similar to WaveNet [2] and Parallel WaveNet [9], but our convolutions have 3 taps and are not causal.”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, background): “Synthesis of high-dimensional data typically takes multiple orders of magnitude longer with AR models; see (Kingma et al., 2016; Oord et al., 2017) for evidence.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “An alternative replacement for the codebook loss described in (Van Den Oord et al., 2017) is to use an EMA update which empirically shows faster training and convergence speed.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, neutral): “There are other waveform models, which often require auxiliary losses for training, such as ﬂow-based models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng et al., 2020), and generative adversarial network (GAN) based models…”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “There has been a plethora of research into non-autoregressive models for audio generation, including normalizing flows such as inverse autoregressive flows (Oord et al., 2018; Ping et al., 2019), generative flows (Prenger et al.”
Measuring the Algorithmic Efficiency of Neural Networks (2020, neutral): “…Jia et al., 2018] Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L., et al. (2018 6, 10, 11, 16 [van den Oord et al., 2017] van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., van den Driessche, G., Lockhart, E., Cobo,…”
The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “…x t with t = 1 , . . . , T , they model the distribution as a product of conditionals: AR models can generate realistic speech signals, and despite the potentially high cost of sampling (each timestep is produced sequentially), these models are now used in practice for text-to-speech (TTS) [45].”
Blockwise Parallel Decoding for Deep Autoregressive Models (2018, neutral): “While advances in this direction have allowed for signiﬁcantly faster training, outputs are still generated one token at a time during inference, posing a substantial challenge for many practical applications (Oord et al., 2017).”
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, methodology): “[28] A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. C. Cobo, F. Stimberg, N. Casagrande, D. Grewe, S. Noury, S. Dieleman, E. Elsen, N. Kalchbrenner, H. Zen, A. Graves, H. King, T. Walters, D. Belov, and D. Hassabis, “Parallel Wav”
Masked Language Model Scoring (2019, methodology): “This is reminiscent of distilling an autoregressive teacher to a parallel student, as in the case of WaveNet (Oord et al., 2018).”