MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

By Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, Yi-Hsuan Yang. 2017

643 citations recorded by the source corpus.

Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative adversarial networks (GANs). The three models, which differ in the underlying assumptions and accordingly the network architectures, are referred to as the jamming model, the composer model and the hybrid model. We trained the proposed models on a dataset of over one hundred thousand bars of rock music and applied them to generate piano-rolls of five tracks: bass, drums, guitar, piano and strings. A few intra-track and inter-track objective metrics are also proposed to evaluate the generative results, in addition to a subjective user study. We show that our models can generate coherent music of four bars right from scratch (i.e. without human inputs). We also extend our models to human-AI cooperative music generation: given a specific track composed by human, we can generate four additional tracks to accompany it. All code, the dataset and the rendered audio samples are available at https://salu133445.github.io/musegan/.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Music Transformer: Generating Music with Long-Term Structure (2018, background): “Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (e.g., Dong et al., 2018) or as orderless NADEs (Uria et al., 2014; 2016) (e.g., Huang et al., 2017).”
Jukebox: A Generative Model for Music (2020, background): “, 2017) and MuseGAN (Dong et al., 2018) which use generative adversarial networks, MusicVAE (Roberts et al.”
Pop Music Transformer: Generating Music with Rhythm and Harmony (2020, methodology): “It is convenient for such approaches to learn the rhythmic structure of music (as shown by [ Dong et al. , 2018 ] for example), as the beats are always same time steps apart in such matrices.”
A Continuous Time Framework for Discrete Denoising Models (2022, methodology): “[29] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang.”
LARA: Attribute-to-feature Adversarial Learning for New-item Recommendation (2020, background): “Moreover, it has already gained huge success in many tasks, such as image generation [22, 40], natural language generation [15, 34, 41], and music generation [9, 10].”
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training (2019, methodology): “Recent work [21–23] attempts multi-instrumental music generation from scratch, but these methods are limited to generating ﬁxed lengths, unlike our method which can generate arbitrarily-long sequences.”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “Building machines that can compose music like human beings is one of the most exciting tasks in multimedia and artificial intelligence [5, 7, 13, 18, 26, 27, 31, 44, 45].”
Modeling Self-Repetition in Music Generation using Generative Adversarial Networks (2019, methodology): “Other Related Works: Dong et al. (2018) and Yang et al. (2017) propose GAN based methods for music generation, with latter using a 2D convolutional discriminator on sequence of generated bars.”
A Hierarchical Recurrent Neural Network for Symbolic Melody Generation (2017, neutral): “titrack music generation were proposed [18].”
MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer (2018, background): “reconstruction performance. GANs, while very powerful, are notoriously difﬁcult to train and have generally not been applied to sequential data. However, Mogren [25], Yang et al. [34] and Dong et al. [10] have recently shown the efﬁcacy of CNNbased GANs for music composition. Yu et al. [35] were the”
Learning a Latent Space of Multitrack Measures (2018, methodology): “MuseGAN [10] is based on generative adversarial networks (GANs) and is capable of modeling multiple instruments over multiple bars.”
PIANOTREE VAE: Structured Representation Learning for Polyphonic Music (2020, methodology): “Piano-roll and its variations [7, 20–22] view polyphonic music as 3-D (one-hot) tensors, in which the first two dimensions denote time and pitch and the third dimension indicates whether the token is an onset, sustain or rest.”
Generative Adversarial Networks (GANs) (2020, background): “[211] proposed a GANs based framework for symbolic multi-track music generation.”