Simple and Controllable Music Generation

By Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre D'efossez. 2023

745 citations recorded by the source corpus.

We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shiliang Zhang
2. Gabriel Synnaeve
3. Yi Yang
4. Dongchao Yang
5. Rongjie Huang
6. Yaron Lipman
7. Yossi Adi
8. Songxiang Liu

In Signal Processing

1. Dongchao Yang
2. Yossi Adi
3. Rongjie Huang
4. Bowen Shi
5. Gabriel Synnaeve
6. Tal Remez
7. Apoorv Vyas
8. Jiang Bian

Citations in the corpus

• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “Music generative models can create music in various styles using a short text description (Schneider et al., 2023; Huang et al., 2023a; Agostinelli et al., 2023; Copet et al., 2023) (Li et al., 2023).”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “From the MOS evaluation performance, we can see that MusicGen is better than our current models.”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “Recent advancesinTTMareevidentinmodelslikeMusicLM[8], Noise2Music[33],MusicGen[34],andMeLoDy[13].”
• JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, neutral): “Others employ inefficient autoregressive designs to sequentially predict each music token [3], or adopt multiple models to cascade-generate the music, resulting in low computational efficiency [4], [6].”
• Discrete Flow Matching (2024, background): “…as, code (Roziere et al., 2023; Li et al., 2023; Chen et al., 2021), biology (Zhang et al., 2024; Ferruz and Höcker, 2022; Madani et al., 2023), math (Romera-Paredes et al., 2024; Imani et al., 2023; Ahn et al., 2024), audio (Kreuk et al., 2022; Copet et al., 2024; Hassid et al., 2024) and more.”
• Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, methodology): “A new more advanced technique is proposed in (Copet et al., 2023), called the delay pattern , but its implementation requires multiple heads and a latent space of higher dimensionality per model.”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “(2) In the second category, discrete codec tokens are employed as audio representations and generated by diffusion models (Yang et al., 2023) or autoregressive language models (Kreuk et al., 2023; Borsos et al., 2023; Copet et al., 2023; Wang et al., 2023a).”
• Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “Recent work has shown the potential of large-scale models for music infilling (Huang et al., 2019a,b) and generation (Agostinelli et al., 2023; Copet et al., 2023; Ren et al., 2020).”
• How I Perceive It: Human Memory-Augmented Analogical Reasoning for Machine Visual Interpretation (2025, unknown): “The animation is then combined with the recomposed glitch sound effects, originally generated by MusicGen [Copet et al. 2023], to create a video.”