Music transcription modelling and composition using deep learning

By Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, Iryna Korshunova. 2016

190 citations recorded by the source corpus.

We apply deep learning methods, specifically long short-term memory (LSTM) networks, to music transcription modelling and composition. We build and train LSTM networks using approximately 23,000 music transcriptions expressed with a high-level vocabulary (ABC notation), and use them to generate new transcriptions. Our practical aim is to create music transcription models useful in particular contexts of music composition. We present results from three perspectives: 1) at the population level, comparing descriptive statistics of the set of training transcriptions and generated transcriptions; 2) at the individual level, examining how a generated transcription reflects the conventions of a music practice in the training transcriptions (Celtic folk); 3) at the application level, using the system for idea generation in music composition. We make our datasets, software and sound examples open and available: \url{this https URL}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

In Signal Processing

Citations in the corpus

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Often, this involves training the RNN to predict the next note in a monophonic melody; however, as mentioned above, the melodies generated by this model tend to wander and lack musical structure.”
MuseGAN: Symbolic-domain Music Generation and Accompaniment with Multi-track Sequential Generative Adversarial Networks (2017, methodology): “Many are based on RNNs, such as DeepBach (Hadjeres and Pachet 2016), MelodyRNN (Waite et al. 2016), folk-rnn (Sturm et al. 2016) and Song from PI (Chu, Urtasun, and Fidler 2016).”
This time with feeling: learning expressive musical performance (2018, background): “Lattner et al [22], Boulanger-Lewandowski et al [3], Bretan et al [5], Herremans et al [17], Roberts et al [35], Sturm [37], to name only a few.”
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment (2017, methodology): “Many of them used RNNs to generate music of different formats, including monophonic melodies (Sturm et al. 2016) and four-voice chorales (Hadjeres, Pachet, and Nielsen 2017).”
Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints (2016, background): “For restricted genres or representations such as monophonic folk melodies (Sturm et al. 2016), symbolic chord sequences, or drum tracks (Choi et al. 2016), and even in polyphonic music with clearly defined melodic, voices such as Bach chorales (Boulanger-Lewandowski et al. 2012a; Hadjeres and…”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, background): “We note that, while the incorporation of the bars is not new in the literature of automatic music composition [6, 20, 38], it represents a brand new attempt in the context of Transformer-based music modeling.”
Tuning Recurrent Neural Networks with Reinforcement Learning (2016, background): “Given a trained Note RNN, the goal is to teach it concepts about music theory, while still maintaining the information about typical melodies originally learned from data.”
Artificial intelligence in the creative industries: a review (2020, methodology): “Modelling music creativity is often achieved using Long Short-Term Memory (LSTM), a special type of RNN architecture [39] (e.”