Song From PI: A Musically Plausible Network for Pop Music Generation

By Hang Chu, Raquel Urtasun, Sanja Fidler. 2016

140 citations recorded by the source corpus.

We present a novel framework for generating pop music. Our model is a hierarchical Recurrent Neural Network, where the layers and the structure of the hierarchy encode our prior knowledge about how pop music is composed. In particular, the bottom layers generate the melody, while the higher levels produce the drums and chords. We conduct several human studies that show strong preference of our generated music over that produced by the recent method by Google. We additionally show two applications of our framework: neural dancing and karaoke, as well as neural story singing.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Some authors have experimented with encoding musical structure into a hierarchical RNN with layers dedicated to generated the melody, drums, and chords (Chu et al., 2016).”
MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation (2017, methodology): “However, it requires prior knowledge of the musical scale and some profiles of the melody to be generated [7], which is not needed in many other models, including MidiNet.”
MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer (2018, background): “ Machines to simultaneously model the temporal structure of music, as well as the harmony between notes that are played at the same time, thus being capable of generating polyphonic music. Chu et al. [7] use domain knowledge to model a hierarchical RNN architecture that produces multi-track polyphon”
MuseGAN: Symbolic-domain Music Generation and Accompaniment with Multi-track Sequential Generative Adversarial Networks (2017, methodology): “First, because there is no clear way to identify which track plays the melody and which plays the chords (Raffel and Ellis 2016), we cannot categorize the piano-rolls into melody, rhythm and drum tracks as some previous work did (Chu, Urtasun, and Fidler 2016) and (Yang, Chou, and Yang 2017).”
A Hierarchical Recurrent Neural Network for Symbolic Melody Generation (2017, methodology): “A 4-layer LSTM is employed to produce the key, press, chord and drum of pop music seperately [5].”
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment (2017, methodology): “Song from PI (Chu, Urtasun, and Fidler 2017) were able to generate a lead sheet (i.e. a track of melody and a track of chord tags) with an additional monophonic drums track by using hierarchical RNNs to co-ordinate the three tracks.”
Learning a Latent Space of Multitrack Measures (2018, methodology): “…polyphonic piano music where a user can control several style parameters, a model from Roy et al. [35] that generates variations on lead sheets, and Song from PI [8] which generates melody, chord, and drum tracks using a hierarchical recurrent network combined with hand-engineered features.”