SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

By Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose M. R. Sotelo, Aaron C. Courville, Yoshua Bengio. 2016

630 citations recorded by the source corpus.

In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Generating Long Sequences with Sparse Transformers (2019, methodology): “…in modeling natural language (Jozefowicz et al., 2016) (Rad-ford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al., 2016) (Menick & Kalchbrenner, 2018) (Salimans et al., 2017) (Reed et al., 2017) (Chen et al., 2017).”
Jukebox: A Generative Model for Music (2020, methodology): “SampleRNN (Mehri et al., 2017) uses a multi-scale, hierarchical recurrent neural network with convolutional upsampling to model long-range complex structures.”
Efficient Neural Audio Synthesis (2018, background): “Copyright 2018 by the author(s). ages (van den Oord et al., 2016b; Reed et al., 2017) and videos (Kalchbrenner et al., 2017) and speech and music (van den Oord et al., 2016a; Mehri et al., 2016; Simon & Oore, 2017; Engel et al., 2017).”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “Most of previous waveform models focus on audio synthesis with informative local conditioner (e.g., mel spectrogram or aligned linguistic features), with only a few exceptions in unconditional generation (Mehri et al., 2017; Donahue et al., 2019).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “Autoregressive models, in particular, have been popular for raw audio generation thanks to their tractable likelihoods, simple inference procedures, and high fidelity samples (Oord et al., 2016; Mehri et al., 2017; Kalchbrenner et al., 2018; Song et al., 2019; Valin & Skoglund, 2019).”
The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “For SampleRNN, this requires adding more tiers .”
Tacotron: Towards End-to-End Speech Synthesis (2017, methodology): “It could be used to predict alternative targets such as vocoder parameters, or as a WaveNetlike neural vocoder [9, 13, 10] that synthesizes waveform samples directly.”
Neural Speech Synthesis with Transformer Network (2018, methodology): “However, acoustic parameters are still intermediate for audios, thus Char2Wav is not a really end-to-end TTS model, and their seq2seq and SampleRNN models need to be separately pre-trained, while Tacotron, proposed by Wang et al.(2017), is an end-to-end generative text-to-speech model, which can be ”
Deep Voice: Real-time Neural Text-to-Speech (2017, background): “Most recently, there has been a lot of work in parametric audio synthesis, notably WaveNet, SampleRNN, and Char2Wav (van den Oord et al., 2016; Mehri et al., 2016; Sotelo et al., 2017).”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “SampleRNN [11] uses recurrent networks in a similar fashion, but it relies on previous samples at different scales.”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “To mitigate this problem, audio generation is typically divided into two stages: 1) predicting audio conditioned on some intermediate representation such as mel-spectrograms [23, 27, 19, 29] and 2) predicting the intermediate representation given some conditioning information, such as text [34, 33].”
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “Autoregressive models, like WaveNet, generate audio samples sequentially, conditioning each new sample on all previously generated ones (Mehri et al., 2016; Kalchbrenner et al., 2018; Valin and Skoglund, 2019).”
A Survey on Neural Speech Synthesis (2021, methodology): “As can be seen, TTS models that use RNNbased autoregressive models [376, 297, 227, 357] are slow in both training and inference, with O(N) computation, where N is the sequence length.”
Neural Voice Cloning with a Few Samples (2018, neutral): “, 2016a], SampleRNN [Mehri et al., 2016], Char2Wav [Sotelo et al.”
Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, background): “…the problem of synthesizing speech with neural networks, including Deep Voice 1 (Arık et al., 2017), Deep Voice 2 (Arık et al., 2017), Tacotron (Wang et al., 2017), Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2017), SampleRNN (Mehri et al., 2017), and WaveNet (Oord et al., 2016).”
Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “…a variety of subcomponents, including duration prediction (Zen et al., 2016), fundamental frequency prediction (Ronanki et al., 2016), acoustic modeling (Zen and Sak, 2015), and more recently autoregressive sample-by-sample audio waveform generation (e.g., Oord et al., 2016; Mehri et al., 2016).”
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “The neural network based vocoders, such as WaveNet (Oord et al., 2016) and SampleRNN (Mehri et al., 2017), play a very important role in recent advances of speech synthesis.”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “Autoregressive waveform models–such as WaveNet (Oord et al., 2016), SampleRNN (Mehri et al., 2016), and WaveRNN (Kalchbrenner et al., 2018)–avoid these issues by generating the waveform a single sample at a time.”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “SampleRNN (Mehri et al., 2016) is an alternative architecture to perform unconditional waveform generation which explicitly models raw audio at different temporal resolutions using multi-scale recurrent neural networks.”