Efficient Neural Audio Synthesis

By Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, Koray Kavukcuoglu. 2018

936 citations recorded by the source corpus.

Sequential models achieve state-of-the-art results in audio, visual and textual domains with respect to both estimating the data distribution and generating high-quality samples. Efficient sampling for this class of models has however remained an elusive problem. With a focus on text-to-speech synthesis, we describe a set of general techniques for reducing sampling time while maintaining high output quality. We first describe a single-layer recurrent neural network, the WaveRNN, with a dual softmax layer that matches the quality of the state-of-the-art WaveNet model. The compact form of the network makes it possible to generate 24kHz 16-bit audio 4x faster than real time on a GPU. Second, we apply a weight pruning technique to reduce the number of weights in the WaveRNN. We find that, for a constant number of parameters, large sparse networks perform better than small dense networks and this relationship holds for sparsity levels beyond 96%. The small number of weights in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile CPU in real time. Finally, we propose a new generation scheme based on subscaling that folds a long sequence into a batch of shorter sequences and allows one to generate multiple samples at once. The Subscale WaveRNN produces 16 samples per step without loss of quality and offers an orthogonal method for increasing sampling efficiency.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yoshua Bengio
5. Quoc V. Le
6. Mohit Bansal
7. Yu Zhang
8. Jianfeng Gao

In Generative Models

1. Jonathan Ho
2. Stefano Ermon
3. Wei Ping
4. Mohammad Norouzi
5. Arash Vahdat
6. Yoshua Bengio
7. Pieter Abbeel
8. Xu Tan

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Junichi Yamagishi
4. Tie-Yan Liu
5. Frank K. Soong
6. Lei He
7. Jinyu Li
8. Yu Zhang

Citations in the corpus

• Denoising Diffusion Probabilistic Models (2020, background): “Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that ha”
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, neutral): “Similar findings hold for large but sparse audio synthesis models (Kalchbrenner et al., 2018) and convolutional models for computer vision (Zhu & Gupta, 2018; Elsen et al.”
• The State of Sparsity in Deep Neural Networks (2019, background): “…this property has been leveraged to significantly reduce the cost associated with the deployment of deep neural networks, and to enable the deployment of state-of-the-art models in severely resource constrained environments (Theis et al., 2018; Kalchbrenner et al., 2018; Valin & Skoglund, 2018).”
• Jukebox: A Generative Model for Music (2020, methodology): “WaveRNN (Kalchbrenner et al., 2018) uses recurrent neural networks that operate separately on the most significant and the least significant bytes, which can be efficiently deployed in mobile devices while having comparable audio quality to WaveNet.”
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “First, with a TTS model, unshuffled and shuffled sentences are forwarded through a WaveRNN vocoder (Kalchbrenner et al., 2018) trained on the LJSpeech dataset7 using the Mozilla TTS toolkit8.”
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “WaveRNN (Kalchbrenner et al., 2018) improves sampling speed for autoregressive models for audio via sparsity and other engineering considerations, some of which may apply to flow models as well.”
• Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “As of this writing, the fastest auto-regressive network is [22], which uses a variety of techniques to speed up an auto-regressive RNN.”
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, background): “We think of the ordering as the two-dimensional analogue of the one-dimensional subscale ordering introduced in Kalchbrenner et al. (2018).”
• Noise Estimation for Generative Diffusion Models (2021, methodology): “Subsequently, in [2] a generative neural diffusion process based on score matching was applied to speech generation, obtaining state of the art results in comparison to well-established methods, such as Wavenet [23], Wavernn [14] and GANTTS [1].”
• DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “In previous work, likelihood-based models, including autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2018; Mehri et al., 2017) and flow-based models (Ping et al., 2020; Prenger et al., 2019; Kim et al., 2019), have predominated in audio synthesis because of the simple training…”
• WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “We chose a female speaker in the training dataset, who had also been used in (Oord et al., 2016; Kalchbrenner et al., 2018; Shen et al., 2018; Bikowski et al., 2020), for evaluation.”
• The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “Recent successful attempts rely on autoregressive (AR) models: WaveNet [43], VRNN [10], WaveRNN [23] and SampleRNN [31] predict digital waveforms one timestep at a time.”
• Blockwise Parallel Decoding for Deep Autoregressive Models (2018, methodology): “Some general-purpose methods include probability density distillation (Oord et al., 2017), subscaling (Kalchbrenner et al., 2018), and decomposing the problem into the autoregressive generation of a short sequence of discrete latent variables followed by a parallel generation step conditioned on…”
• SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “Other approaches involve lightweight and sparse models [13] and networks mimicking the fast Fourier transform as part of the model [7], [14].”
• What is the State of Neural Network Pruning? (2020, neutral): “Kalchbrenner et al. (2018) obtain a similar result for audio synthesis, as do Gray et al. (2017) for a variety of additional tasks across various domains.”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “Autoregressive models, like WaveNet, generate audio samples sequentially, conditioning each new sample on all previously generated ones (Mehri et al., 2016; Kalchbrenner et al., 2018; Valin and Skoglund, 2019).”