Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

By Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, Karen Simonyan. 2017

781 citations recorded by the source corpus.

Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “…(Hiller Jr & Isaacson, 1957; Moorer, 1972; Hadjeres et al., 2017; Huang et al., 2017), digital vocoders generating a singer’s voice (Bonada & Serra, 2007; Saino et al., 2006; Blaauw & Bonada, 2017) and also synthesizers producing timbres for various musical instruments (Engel et al., 2017; 2019).”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “On the other hand these same technologies also enable assistive technologies like text to speech [58], new art forms [59, 60], and new design technologies [61].”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “Latent collapse can also be avoided in deterministic autoencoders, such as [64], which coupled a convolutional encoder to a powerful autoregressive WaveNet decoder [18] to learn a latent representation of music audio consisting of isolated notes from a variety of instruments.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “In a typical deterministic autoencoder, the information content of the learnt representation is limited only by the capacity of the encoder and decoder, because the representation is real-valued (if they are very nonlinear, they could compress a lot of information into even a single scalar value) [1”
Neural Voice Cloning with a Few Samples (2018, background): “, 2017), audio synthesis (Wang et al., 2017; Engel et al., 2017; Arik et al., 2017a), and language modeling (Jozefowicz et al.”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “Model Size: Table B.6, compares parameter counts for the DDSP models and comparable models including GANSynth (Engel et al., 2019), WaveRNN (Hantrakul et al., 2019), and a WaveNet Autoencoder (Engel et al., 2017).”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “These methods have produced state-of-the-art results in text-to-speech synthesis (Sotelo et al., 2017; Shen et al., 2018; Ping et al., 2017) and other audio generation tasks (Engel et al., 2017).”
It's Raw! Audio Generation with State-Space Models (2022, methodology): “(iii) Autoencoders: WaveNet Autoencoder [11] and WaveVAE [31], which use WaveNets in their encoders.”
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, methodology): “WaveNet has also been adapted by Engel et al. (2017) for their NSynth model, addressing the representation learning task.”
Adversarial Audio Synthesis (2018, background): “Engel et al. (2017) pose WaveNet as an autoencoder to generate musical instrument sounds.”
High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “The latter provides state-of-the-art results for a dataset of single note recordings from various musical instruments (NSynth, Engel et al., 2017) by training GANs to generate invertible spectrograms of the notes.”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, background): “This idea has been explored extensively in literature [8, 12, 16, 19, 50, 52].”
GANSynth: Adversarial Neural Audio Synthesis (2019, methodology): “NSynth is a difﬁcult dataset composed of highly diverse timbres and pitches, but it is also highly structured with labels for pitch, velocity, instrument, and acoustic qualities (Liu et al., 2015; Engel et al., 2017).”
Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks (2018, background): “When a circular normal distribution is assumed for the phase, the log-likelihood loss is proportional to cos(φ(STFT(s)) − φ(STFT(ŝ))) [16].”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, methodology): “Our interest in deep latent variable models primarily comes from their increasing use in creative applications of machine learning (Carter & Nielsen, 2017; Ha & Eck, 2018; Engel et al., 2017a).”
COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations (2020, methodology): “Finally, for MIC, we use the NSynth dataset (Engel et al., 2017) which consists of more than 300k sound samples organised in 10 instrument families.”