WaveNet: A Generative Model for Raw Audio
By Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, Koray Kavukcuoglu. 2016
8,319 citations recorded by the source corpus.
This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Stefano Ermon
- Arash Vahdat
- Yoshua Bengio
- Diederik P. Kingma
- Jiaming Song
- Jonathan Ho
- Jun-Yan Zhu
- Aaron C. Courville
In Speech & Audio
In Signal Processing
Citations in the corpus
- Neural Machine Translation in Linear Time (2016, methodology): “The scheme is repeated multiple times in the network always starting from a dilation rate of 1 (van den Oord et al., 2016a; Kalchbrenner et al., 2016b).”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “To adapt the Gated CNN for bidirectional language modeling, we closely followed the publicly available ConvSeq2Seq implementation, 4 modi-fied to support causal convolutions (van den Oord et al., 2016) for both the forward and backward directions.”
- Denoising Diffusion Probabilistic Models (2020, background): “…models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [12, 25, 3, 55, 35, 23, 10, 30, 41, 54, 24, 31, 42], and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable to those of…”
- Generating Long Sequences with Sparse Transformers (2019, methodology): “To address this, WaveNet (Van Den Oord et al., 2016) introduced dilated convolutions, which allowed the network to model long-range dependencies in a logarithmic number of layers.”
- KERMIT: Generative Insertion-Based Modeling for Sequences (2019, background): “…many conditional generation applications, including machine translation (Bahdanau et al., 2015; Luong et al., 2015), speech recognition (Chan et al., 2016; Bahdanau et al., 2016), speech synthesis (Oord et al., 2016; Wang et al., 2017) and image captioning (Vinyals et al., 2015; Xu et al., 2015).”
- Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “We show the Compressive Transformer works not only for language, but can also model the waveform of high-frequency speech with a trend of lower likelihood than the TransformerXL and Wavenet (Oord et al., 2016) when trained over 400,000 steps.”
- Evaluating Large Language Models Trained on Code (2021, background): “…(Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal…”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “For years, the fields have been continuously reporting new state of the art results using varieties of model architectures for computer vision tasks [57, 58, 7], for natural language understanding tasks [59, 60, 61], for speech recognition and synthesis tasks [62, 63, 64, 65, 66].”
- Longformer: The Long-Document Transformer (2020, background): “This is analogues to dilated CNNs (van den Oord et al., 2016) where the window has gaps of size dilation d (Fig.”
- Are we done with ImageNet? (2020, background): “For example, progress on ImageNet due to architecture design [2, 3, 4] or optimization [5] has yielded corresponding gains on other tasks and modalities [6, 7, 8].”
- On the Relationship between Self-Attention and Convolutional Layers (2019, background): “Convolutional layers acting on sequences are commonly used in the literature for text (Kim, 2014), as well as audio (van den Oord et al., 2016) and time series (Franceschi et al., 2019).”
- Stand-Alone Self-Attention in Vision Models (2019, background): “An emerging theme of augmenting convolution models with self-attention has yielded gains in several vision tasks.”
- Non-local Neural Networks (2017, background): “, non-recurrent) networks for modeling sequences in speech and language [36, 54, 15].”
- Jukebox: A Generative Model for Music (2020, methodology): “WaveNet (Oord et al., 2016) performs autoregressive sample-by-sample probabilistic modeling of raw waveform using a series of dilated convolutions to exponentially increase the context length.”
- Representation Learning with Contrastive Predictive Coding (2018, background): “More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
- Neural Discrete Representation Learning (2017, methodology): “While samples drawn from even the best speech models like the original WaveNet [37] sound like babbling , samples from VQ-VAE contain clear words and part-sentences (see samples linked above).”
- Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, background): “, 2016), speech synthesis (Oord et al., 2016; 2018) and image captioning (Vinyals et al.”