Noise2Music: Text-conditioned Music Generation with Diffusion Models

By Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Havnø Frank and 4 more. 2023

263 citations recorded by the source corpus.

We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era, but goes beyond to ground fine-grained semantics of the prompt. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “…modality coverage, and quantities compared to previous large scale audio generative model studies (Yang et al., 2023b; Borsos et al., 2023; Wang et al., 2023a; Liu et al., 2023c), which leverage datasets ranging between 10K to 100K hours containing mostly speech from a single domain…”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “With the recent explosion of generative image models [47, 40, 48, 17, 52, 53, 70, 15], there has been increasing interest in training similar generative models for other modalities such as audio [44, 10, 5, 31, 2, 25], video [24, 57, 23], and 3D assets [26, 28, 45, 41].”
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, unknown): “As shown in Table II, we compare the performance of JEN-1 with other state-of-the-art methods, including Riffu-sion [33], and Mousai [23], MusicLM [3], MusicGen [4], Noise2Music [6].”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “Besides, diffusion models also show great success in text-to-video generation [18, 47, 52], 3D Avatar generation [39, 53], image to image translation [37], image manipulation [4, 26], music generation [23], and even drug discovery [55].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based generation has been extended to other modalities including audio [35, 17, 28, 70], video [75, 91, 85, 25], and 4D fields [76].”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “Furthermore, MusicLM [3] and Noise2Music [17] generate music that matches the semantic information in the input text.”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, unknown): “Recent advancesinTTMareevidentinmodelslikeMusicLM[8], Noise2Music[33],MusicGen[34],andMeLoDy[13].”
Simple and Controllable Music Generation (2023, background): “Schneider et al. [2023], Huang et al. [2023b], Maina [2023], Forsgren and Martiros proposed using a latent diffusion model for the task of text-to-music.”