Encoding Musical Style with Transformer Autoencoders

By Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, Jesse Engel. 2019

104 citations recorded by the source corpus.

We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across time to obtain a global representation of style from a given performance. We show it is possible to combine this global representation with other temporally distributed embeddings, enabling improved control over the separate aspects of performance style and melody. Empirically, we demonstrate the effectiveness of our method on various music generation tasks on the MAESTRO dataset and a YouTube dataset with 10,000+ hours of piano performances, where we achieve improvements in terms of log-likelihood and mean listening scores as compared to baselines.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yu-Siang Huang
2. Yi-Hsuan Yang
3. Curtis Hawthorne
4. Jesse Engel
5. Ian Simon
6. Gautam Mittal
7. Derya Soydaner
8. Udeepa Meepegama

In Generative Models

1. Yu-Siang Huang
2. Yi-Hsuan Yang
3. Jesse Engel
4. Ian Simon
5. Curtis Hawthorne
6. Gautam Mittal
7. Derya Soydaner
8. Arash Vahdat

In Multimodal Learning

1. Yu-Siang Huang
2. Yi-Hsuan Yang
3. Jesse Engel
4. Ian Simon
5. Curtis Hawthorne
6. Derya Soydaner

In Signal Processing

1. Yu-Siang Huang
2. Yi-Hsuan Yang
3. Jesse Engel
4. Curtis Hawthorne
5. Ian Simon
6. Gautam Mittal
7. Derya Soydaner
8. Adam Roberts

Citations in the corpus

• Symbolic Music Generation with Diffusion Models (2021, methodology): “Inspired by the statistical similarity evaluation described in [27], we evaluate our models with a modified framewise Overlapping Area ( OA ) metric.”
• Pop Music Transformer: Generating Music with Rhythm and Harmony (2020, methodology): “Among the approaches that have been studied, sequence models, which consider music as a language , stand out in recent work [ Oore et al. , 2018; Huang et al. , 2019; Payne, 2019; Choi et al. , 2019] great potential.”
• Generative Pretraining for Paraphrase Evaluation (2021, methodology): “The musical style Transformer autoencoder [33] uses a similar Transformerbased style transfer architecture to conditionally generate new music in controllable styles.”
• Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Transformer Autoencoder is a sequential autoencoder for conditional music generation [123].”
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “ply follows the MIDI-like event representation proposed by [30] to set up the “vocabulary” for music. As shown in Table 1, the MIDI-like representation, for the case of modeling classical piano music [9, 23], uses Note-On events to indicate the action of hitting a specific key of the piano keyboard,”