Symbolic Music Generation with Diffusion Models

By Gautam Mittal, Jesse Engel, Curtis Hawthorne, Ian Simon. 2021

234 citations recorded by the source corpus.

Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. Our method is non-autoregressive and learns to generate sequences of latent embeddings through the reverse process and offers parallel generation with a constant number of iterative refinement steps. We apply this technique to modeling symbolic music and show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Generative models trained on large amounts of unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67].”
Score-based Generative Modeling in Latent Space (2021, methodology): “In principle LSGM could be used to model, for example, language [96, 97], music [98, 10], or molecules [99, 100].”
On Fast Sampling of Diffusion Probabilistic Models (2021, background): “, 2021), music synthesis (Liu et al., 2021; Mittal et al., 2021), 3-D point cloud generation (Luo & Hu, 2021; Zhou et al.”
Gotta Go Fast When Generating Data with Score-Based Models (2021, neutral): “…al., 2020, Song et al., 2020a, Piché-Taillefer, 2021] have been very successful at generating data from various modalities, such as images [Ho et al., 2020, Song et al., 2020a], audio [Chen et al., 2020, Kong et al., 2020, Mittal et al., 2021, Kameoka et al., 2020], and graphs [Niu et al., 2020].”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “Some works [29, 27] have dealt with discrete data by embedding it in continuous space and leveraging Gaussian diffusion, but have not applied this to text.”
Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations (2022, background): “, 2021b;a; Dhariwal & Nichol, 2021), audio (Chen et al., 2021; Kong et al., 2021; Jeong et al., 2021; Mittal et al., 2021), and point clouds (Cai et al.”
Post-Training Quantization on Diffusion Models (2022, unknown): “nomenal success in various generative tasks, such as images [10, 36, 23], audio [20], video [31], and graphs [24].”
A Continuous Time Framework for Discrete Denoising Models (2022, methodology): “We parameterize p θ 0 | t with a transformer architecture [31] and train using a conditional form of L CT targeting the conditional distribution of the final 14 bars (224 time steps) given the first 2 bars of the song.”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, unknown): “In music generation tasks, [23] introduced a technique for training diffusion models on sequential data and applied it to model symbolic music.”
DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs (2022, background): “Diffusion models have obtained state-of-the-art results in several important domains, including image synthesis(Ho et al., 2020), audio synthesis(Kong et al., 2021; Chen et al., 2021; Popov et al., 2021), graphs(Niu et al., 2020), and symbolic music generation (Mittal et al., 2021).”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[92] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon.”
Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives (2021, background): “…translation (Sasaki et al., 2021), speech synthesis (Chen et al., 2020, 2021; Kong et al., 2021; Popov et al., 2021), symbolic music generation (Mittal et al., 2021), natural language generation (Hoogeboom et al., 2021; Austin et al., 2021), etc. DDPM is unique in that it considers an…”
On Distillation of Guided Diffusion Models (2022, methodology): “Latent diffusion models (LDMs) [21,24,28,35] increase the training and inference efficiency of diffusion models (directly learned on the pixel-space) by modeling images in the latent space of a pre-trained regularized autoencoder, where the latent representations are usually of lower dimensionality ”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “, 2021), and music synthesis (Mittal et al., 2021), image editing (Meng et al.”
CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation (2021, methodology): “To utilize diffusion models for imputation, previous studies [12, 15, 16] approximated the conditional reverse process pθ(x t−1 | x t ,x 0 ) with the reverse process in Eq.”
Flexible Diffusion Modeling of Long Videos (2022, methodology): “The combination of this diffusion process and a data distribution q ( x 0 , y ) (recalling that x 0 = x ) defines the joint distribution q ( x 0: T , y ) = q ( x 0 , y ) DDPMs work by “inverting” the diffusion process: given values of x t and y a neural network is used to parameterize p θ ( x t − 1 ”