MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation

By Li-Chia Yang, Szu-Yu Chou, Yi-Hsuan Yang. 2017

509 citations recorded by the source corpus.

Most existing neural network models for music generation use recurrent neural networks. However, the recent WaveNet model proposed by DeepMind shows that convolutional neural networks (CNNs) can also generate realistic musical waveforms in the audio domain. Following this light, we investigate using CNNs for generating melody (a series of MIDI notes) one bar after another in the symbolic domain. In addition to the generator, we use a discriminator to learn the distributions of melodies, making it a generative adversarial network (GAN). Moreover, we propose a novel conditional mechanism to exploit available prior knowledge, so that the model can generate melodies either from scratch, by following a chord sequence, or by conditioning on the melody of previous bars (e.g. a priming melody), among other possibilities. The resulting model, named MidiNet, can be expanded to generate music with multiple MIDI channels (i.e. tracks). We conduct a user study to compare the melody of eight-bar long generated by MidiNet and by Google's MelodyRNN models, each time using the same priming melody. Result shows that MidiNet performs comparably with MelodyRNN models in being realistic and pleasant to listen to, yet MidiNet's melodies are reported to be much more interesting.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “…DeepBach (Hadjeres et al., 2017) and Coconet (Huang et al., 2017) which use Gibbs sampling to produce notes in the style of Bach chorals, MidiNet (Yang et al., 2017) and MuseGAN (Dong et al., 2018) which use generative adversarial networks, MusicVAE (Roberts et al., 2018) and HRNN (Wu et al.,…”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, methodology): “GAN for symbolic-domain and music generation which performs comparably against Melody RNN [214].”
MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer (2018, methodology): “gh interpolation and reconstruction performance. GANs, while very powerful, are notoriously difﬁcult to train and have generally not been applied to sequential data. However, Mogren [25], Yang et al. [34] and Dong et al. [10] have recently shown the efﬁcacy of CNNbased GANs for music composition. Yu”
Pop Music Transformer: Generating Music with Rhythm and Harmony (2020, methodology): “Image-based approaches such as MidiNet [ Yang et al. , 2017 ] and MuseGAN [ Dong et al. , 2018 ] use an image-like representation such as the piano roll to represent a score as a matrix of time steps and MIDI pitches, and then use convolution-based operations to generate music.”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “Image-based approaches such as MidiNet [41] and MuseGAN [13] use an imagelike representation such as the piano roll to represent a score as a matrix of time steps and MIDI pitches, and then use convolutionbased operations to generate music.”
Deep Learning Techniques for Music Generation - A Survey (2017, methodology): “Note that this is consonant with the use of this regularization technique to control creativity trade-off in MidiNet (see Section 7.9.1.1).”
Generative Adversarial Networks (GANs) (2020, methodology): “[212] introduced a GANs based model to produce melodies either from scratch, by following a chord sequence, or by conditioning on the melody of previous bars.”
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment (2017, methodology): “Since there is no clear way to identify which track plays the melody and which plays the accompaniment (Raffel and Ellis 2016), we cannot categorize the tracks into melody, rhythm and drum tracks as some prior works did (Chu, Ur-tasun, and Fidler 2017; Yang, Chou, and Yang 2017).”
Generative Adversarial Networks for Electronic Health Records: A Framework for Exploring and Evaluating Methods for Predicting Drug-Induced Laboratory Test Trajectories (2017, background): “While GANs have demonstrated their power in applications such as text-to-image synthesis [13, 18], image-to-image translation [7], video generation [15], and even music generation [17], applications to healthcare remain scarce.”
Machine Learning and Deep Learning frameworks and libraries for large-scale data mining: a survey (2019, background): “…detection (Schlegl et al. 2017), 3D object generation (Wu et al. 2016a) [for instance dental restoration (Hwang et al. 2018)], music generation (Yang et al. 2017), scientific simulations acceleration [for example in astrophysics (Mustafa et al. 2017) or high energy physics (Paganini et al.…”
Statistical parametric speech synthesis using generative adversarial networks under a multi-task learning framework (2017, methodology): “GAN is a powerful generative model that has been successfully used in image generation [13, 18, 19] and other tasks [20, 21].”
Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations (2018, background): “Generative adversarial networks (GANs) have achieved remarkable success within short time for diverse tasks of generating synthetic data, such as images [1, 2, 3, 4], texts [5, 6, 7, 8], and even music [5, 9, 10, 11].”
A Hierarchical Recurrent Neural Network for Symbolic Melody Generation (2017, methodology): “A user study suggested that the RNN with Lookback feature outperforms basic RNN [3] so we also use it in our model1.”
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training (2019, background): “Here, most work represents polyphonic music as a piano roll—a sparse binary matrix of time and pitch—and seeks to generate sequences of individual piano roll timesteps [12, 13] or chunks of timesteps [14].”
Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images (2018, background): “This potentially allows the model to converge to a better local optimum, and achieve better performance.”