The challenge of realistic music generation: modelling raw audio at scale

By Sander Dieleman, Aäron van den Oord, Karen Simonyan. 2018

191 citations recorded by the source corpus.

Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Generating Long Sequences with Sparse Transformers (2019, methodology): “To test the extent to which Sparse Transformers are able to scale to very long contexts, we trained models on the classical music dataset released by (Dieleman et al., 2018).”
Jukebox: A Generative Model for Music (2020, methodology): “To do so, we use a deep residual WaveNet (Xie et al., 2017) followed by an upsampling strided convolution and a layer norm (Ba et al., 2016), and add the output as extra positional information to the embed-dings of the current level.”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “Finally, the encoder and decoder used in this work retains the simplicity and speed of the original VQ-VAE, which means that the proposed method is an attractive solution for situations in which fast, low-overhead encoding and decoding of large images are required.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “Moreover, the neural networks for VAEs should model long-range correlations in data [33, 34, 35], requiring the networks to have large receptive ﬁelds.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “The contrastive task depends on the codebook to represent both positive and negative examples and the diversity loss L d is designed to increase the use of the quantized codebook representations [10].”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “Multilingually pretrained NLP models such as mBERT [2], XLM-R [8] or mT5 [9] brought significant improvements in multilingual language understanding [10, 11].”
Unsupervised Speech Recognition (2021, methodology): “This is augmented by a codebook diversity penalty to encourage the model to use all codebook entries (Dieleman et al., 2018).”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “For example, in variational autoencoders, vector quantization has been used to generate images [31], [32] and music [33], [34].”
High Fidelity Neural Audio Compression (2022, background): “Dieleman et al. (2018); Dhariwal et al. (2020) proposed a hierarchical VQ-VAE based model for learning discrete representation of raw audio, next combined with an auto-regressive model, demonstrating the ability to generate high quality music.”
It's Raw! Audio Generation with State-Space Models (2022, background): “[9], we encourage readers to form their own opinions by referring to the sound examples in our supplementary material.”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, methodology): “We can extend this approach further by stacking encoder models, yielding a hierarchy of progressively more high-level representations (Dieleman et al., 2018), as shown in Figure 2.”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, background): “In Dieleman et al. [11] and van den Oord et al. [51] capture long-range dependencies in waveforms by utilizing a hierarchy of autoencoders.”
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (2018, methodology): “We also updated the model to produce 16-bit output using a mixture of logistics as described in van den Oord et al. (2018).”
Quant GANs: deep generation of financial time series (2019, methodology): “Dieleman et al. (2018)), we leave it as future work to develop these methods.”
Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion (2023, background): “However, like text, music generation has long been a challenging task, as it requires multiple aspects at different levels of abstraction (van den Oord et al., 2016; Dieleman et al., 2018).”