Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion

By Flavio Schneider, Ojasv Kamal, Zhijing Jin, Bernhard Scholkopf. 2023

117 citations recorded by the source corpus.

Recent years have seen the rapid development of large generative models for text; however, much less research has explored the connection between text and another"language"of communication -- music. Music, much like text, can convey emotions, stories, and ideas, and has its own unique structure and syntax. In our work, we bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Mo\^usai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. Moreover, our model features high efficiency, which enables real-time inference on a single consumer GPU with a reasonable speed. Through experiments and property analyses, we show our model's competence over a variety of criteria compared with existing music generation models. Lastly, to promote the open-source culture, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: Codes: https://github.com/archinetai/audio-diffusion-pytorch; music samples for this paper: http://bit.ly/44ozWDH; all music samples for all models: https://bit.ly/audio-diffusion.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “Large scale general-domain models for sound and music generation: Text-to-sound (Kreuk et al., 2022) and text-to-music (Schneider et al., 2023) are the emerging paradigms for general-domain sound and music generation, in contrast to earlier studies that generate finite sound effects (Donahue et…”
Simple and Controllable Music Generation (2023, methodology): “Schneider et al. [2023], Huang et al. [2023b], Maina [2023], Forsgren and Martiros proposed using a latent diffusion model for the task of text-to-music.”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based generation has been extended to other modalities including audio [35, 17, 28, 70], video [75, 91, 85, 25], and 4D fields [76].”