MusicLM: Generating Music From Text

By A. Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi and 3 more. 2023

722 citations recorded by the source corpus.

We introduce MusicLM, a model generating high-fidelity music from text descriptions such as"a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “It sounds sad and soulful, like something you would hear at Sunday services.” for music (Agostinelli et al., 2023).”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “…(with duration), speaker prompt, MIDI singing Text-to-Sound (TT-Sound) (Yang et al., 2023c) textual description sounds Text-to-Music (TT-Music) (Agostinelli et al., 2023) textual description music Audio Edit (A-Edit) ♣♢ (Wang et al., 2023d) textual description, original sounds sounds Speech…”
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Notably, generative models using multi-modal input have emerged as a remarkable innovation [5], [6], [7], [8], [9], [10].”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “In the particular case of text-to-music [Agostinelli et al., 2023] or text-to-speech [Kharitonov et al.”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “We believe this interaction is beneficial when training hierarchical generative models on top of these codes [5, 40, 1], for example to partition the codes into “coarse” tokens (denoting the most significant codes) and “fine” tokens.”
HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, methodology): “In this study, we focus on using audio codec models to help solve generation problems in audio-related works, such as Text-to-Speech (TTS) [1, 2], music generation [3], audio generation [4–6].”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “These methods represent continuous audio waveforms as sequences of tokens from a ﬁnite vocabulary, casting speech generation as a language modeling task.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “…(VQ) acoustic features rather than predicting the commonly-used hand-crafted intermediate acoustic features, such as the mel-spectrogram. method only needs one-stage training, and it is a non-autoregressive model, which is far different from the concurrent work VALL-E [10] and MusicLM [11].”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “AudioLM [4] and MusicLM [1] cast audio synthesis as a language modeling task and leverage a hierarchy of coarse-to-fine units.”
Multimodal Large Language Models: A Survey (2023, background): “Moreover, Google’s MusiclM [59] has gained recognition in the audio multimodal domain, as it can generate music based on text inputs.”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, background): “Large language models (LLMs) have taken the world by storm, showing their ability to generate high-quality text for various tasks like storytelling, serving as chat assistants, and even composing music [1, 11].”
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, methodology): “The overall learning of diffusion models involves a forward diffusion process and a gradual denoising process, each consisting of a sequence of T steps that act as a Markov Chain.”
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, positive): “Recently, this framework has also been applied to generating music from text descriptions [20], accompaniment for singing [21], and text to speech (TTS) [22].”
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, methodology): “First, we conducted pre-training using a diverse corpus consisting of textual prompts and audio samples sourced from WavCaps [24], AudioCaps, ESC [26], Urban-Sound [36], MusicCaps [1], GTZAN [40], and Musical Instruments 4 dataset.”
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “Other than natural language processing, there are also emerging efforts in building foundation models for computer vision [13], music generation [1], biology [23, 25], and speech recognition [36].”
Noise2Music: Text-conditioned Music Generation with Diffusion Models (2023, methodology): “Lastly, we evaluate on MusicCaps (Agostinelli et al., 2023) which consists of 5.5K 10-second clips from Au-dioSet paired with rater written captions.”