LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training

By Chris Donahue, Huanru Henry Mao, Yiting Li, Garrison W. Cottrell, Julian McAuley. 2019

137 citations recorded by the source corpus.

We are interested in the task of generating multi-instrumental music scores. The Transformer architecture has recently shown great promise for the task of piano score generation; here we adapt it to the multi-instrumental setting. Transformers are complex, high-dimensional language models which are capable of capturing long-term structure in sequence data, but require large amounts of data to fit. Their success on piano score generation is partially explained by the large volumes of symbolic data readily available for that domain. We leverage the recently-introduced NES-MDB dataset of four-instrument scores from an early video game sound synthesis chip (the NES), which we find to be well-suited to training with the Transformer architecture. To further improve the performance of our model, we propose a pre-training technique to leverage the information in a large collection of heterogeneous music, namely the Lakh MIDI dataset. Despite differences between the two corpora, we find that this transfer learning procedure improves both quantitative and qualitative performance for our primary task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Luke S. Zettlemoyer
3. Yoshua Bengio
4. Kyunghyun Cho
5. Mohit Bansal
6. Jianfeng Gao
7. Sergey Levine
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Jitendra Malik
5. Trevor Darrell
6. Xiaogang Wang
7. Andrea Vedaldi
8. Andrew Zisserman

In Generative Models

1. Heewoo Jun
2. Ilya Sutskever
3. Prafulla Dhariwal
4. Stefano Ermon
5. Yu-Siang Huang
6. Alec Radford
7. Yoshua Bengio
8. Junbo Jake Zhao

In Multimodal Learning

1. Yu-Siang Huang
2. Heewoo Jun
3. Prafulla Dhariwal
4. Yi-Hsuan Yang
5. Ilya Sutskever
6. Alec Radford
7. Alex Nichol
8. Daniel Cohen-Or

In Signal Processing

1. Yu-Siang Huang
2. Heewoo Jun
3. Ruihan Yang
4. Ilya Sutskever
5. Gus G. Xia
6. Yixiao Zhang
7. Yiyi Zhang
8. Ziyu Wang

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “Generative models for music can also be used for music style transfer, as seen in Midi-VAE (Brunner et al., 2018) which uses a variational autoencoder to transfer styles between classical and jazz music, LakhNES (Donahue et al., 2019) which uses a Transformer architecture to generate chiptune music,”
• PIANOTREE VAE: Structured Representation Learning for Polyphonic Music (2020, methodology): “Models MIDI-like event sequence is first used in deep music generation in performanceRNN [24] and Multi-track Music-VAE [9], and then broadly applied in transformer-based generation [19, 25, 26].”
• Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “The main differences to the representation described in Section C.1 are a more compact encoding of timing, no representation of musical dynamics (for simplicity), and support for multiple tracks (not originally proposed by C´ıfka et al., 2020 but introduced here inspired by Donahue et al., 2019).”
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “However, as the use of Transformer-XL for automatic music composition has been reported elsewhere [12], we omit such an empirical performance comparison between the XL and non-XL versions.”