Jukebox: A Generative Model for Music

By Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever. 2020

984 citations recorded by the source corpus.

We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples at this https URL, along with model weights and code at this https URL

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “, 2016), half-precision Adam statistics (Dhariwal et al., 2020), and half-precision stochastically rounded text encoder weights were used.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, neutral): “…weight update in 32-bit and then apply the following transformation at the end of each optimization step to maintain the optimizer state in 16-bit (Dhariwal et al., 2020): ~mfp16 = ~m max abs(~m) FLOAT16_MAX + where = 10−8 and FLOAT16_MAX = 65504.0 is the largest finite value expressible in FP16.”
Evaluating Large Language Models Trained on Code (2021, background): “, 2021), audio and speech processing (Oord et al., 2016; 2018; Dhariwal et al., 2020; Baevski et al., 2020), biology (Alley et al.”
Zero-Shot Text-to-Image Generation (2021, background): “Specifically, when compute, model size, and data are scaled carefully, autoregressive transformers (Vaswani et al., 2017) have achieved impressive results in several domains such as text (Radford et al., 2019), images (Chen et al., 2020), and audio (Dhariwal et al., 2020).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
Score-based Generative Modeling in Latent Space (2021, neutral): “In principle LSGM could be used to model, for example, language [96, 97], music [98, 10], or molecules [99, 100].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, positive): “For VQ-VAE training, we use random restarts for embeddings, and codebook initialization by copying encoder latents as described in (Dhariwal et al., 2020).”
Symbolic Music Generation with Diffusion Models (2021, background): “In the music domain, these techniques have led to improvements in expressive unconditional and conditional generation of both continuous audio (Engel et al., 2017b; 2019; Dhariwal et al., 2020; Engel et al., 2020), and discrete symbolic music (Huang et al.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “The answer has signiﬁcant practical stakes, because large, compute-intensive autoregressive models (Strubell et al., 2019) are increasingly used for a variety of applications (Oord et al., 2016; Brown et al., 2020; Dhariwal et al., 2020; Chen et al., 2020).”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “, 2019), music (Dhariwal et al., 2020) and speech (Ping et al.”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “The method of tokenizing non-discrete modalities to use standard sequence modeling approaches have been extensively explored with DALL-E for images, Jukebox for Music (Dhariwal et al., 2020) and vq-wav2vec for Speech (Baevski et al., 2019).”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “3, using the VQ-VAE architecture [vdOVK18] with modifications described in [DJP20], including dead code revival.”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “Large scale pre-training has led to signiﬁcant advances in vision (Krizhevsky et al., 2012; Girshick et al., 2014), natural language processing (Devlin et al., 2019; Radford et al., 2018; 2019; Brown et al., 2020) and audio understanding (Van Den Oord et al., 2016; Dhariwal et al., 2020).”
Generating Images with Sparse Representations (2021, background): “While memory-efficient Transformer variants (Child et al., 2019; Kitaev et al., 2020; Choromanski et al., 2021; Dhariwal et al., 2020; Zaheer et al., 2021; Roy et al., 2020) make long-sequence training more practical, care is still required to optimize model hyperparameters against available memory…”
Toward Causal Representation Learning (2021, background): “Fuelled by abundant data and compute, AI has made remarkable advances in a wide range of applications, from image processing and natural language processing [36] to beating human world champions in games, such as chess, poker, and Go [223], improving medical diagnoses [166], and generating music [57”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “Second, as proposed in [34], when a codebook vector has not been assigned any input frame for several batches, we replace it with an input frame randomly sampled within the current batch.”
High Fidelity Neural Audio Compression (2022, methodology): “Dieleman et al. (2018); Dhariwal et al. (2020) proposed a hierarchical VQ-VAE based model for learning discrete representation of raw audio, next combined with an auto-regressive model, demonstrating the ability to generate high quality music.”