Music Transformer: Generating Music with Long-Term Structure

By Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck. 2018

598 citations recorded by the source corpus.

Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “…research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b].”
Generating Long Sequences with Sparse Transformers (2019, methodology): “(Huang et al., 2018) apply Transformers to MIDI generation with an efficient relative attention.”
Rethinking Attention with Performers (2020, background): “, 2018)), music generation (Huang et al., 2019), and bioinformatics (Rives et al.”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “We describe our results in Table 2 and compare it to other recent work on sparse or recurrent attention such as Adaptive Inputs (Baevski and Auli, 2018) and Trans-formerXL (Dai et al., 2019) as well as a local attention with relative position encoding baseline (Huang et al., 2018).”
Reformer: The Efficient Transformer (2020, methodology): “Up to 11 thousand tokens of text in a single example were processed in (Liu et al., 2018) and when processing other modalities, like music (Huang et al., 2018) and images (Parmar et al., 2018), even longer sequences are commonplace.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “For an input sequence of length N , it requires a space complexity of OpN2dq [19, 20, 21] to store the relative position embedding for each token.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Its ability to directly encode long-range interactions and its parallelizability, has led to state-of-the-art performance for various tasks [83,40,26,69,75,25,56].”
Attention Augmented Convolutional Networks (2019, methodology): “Additionally, we enhance the representational power of self-attention over images by extending relative self-attention [37, 18] to two dimensional inputs allowing us to model translation equivari-ance in a principled way.”
Jukebox: A Generative Model for Music (2020, methodology): “…et al., 2018) which use generative adversarial networks, MusicVAE (Roberts et al., 2018) and HRNN (Wu et al., 2019) which use hierarchical recurrent networks, and Music Transformer (Huang et al., 2019a) and MuseNet (Payne, 2019) which use Transformers to au-toregressively predict MIDI note events.”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “…“for free” by examining the weight assigned by the neural network to different regions in the input (Graves, 2013; Raffel et al., 2017; Huang et al., 2018), but this introspection method has been shown to be unreliable (Jain & Wallace, 2019; Serrano & Smith, 2019; Pruthi et al.,…”
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021, methodology): “T5 Bias Though most models use trained or sinusoidal position embeddings, the T5 model of Raffel et al. (2020) uses a relative position method (Shaw et al., 2018; Huang et al., 2019) that adds no position information to word embeddings (as in the previous method).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “A speciﬁc transformer architecture symbolic visualization method (applied to music) picto-rially shows how soft-max attention works [314] .”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “To efﬁciently implement relative position encoding (a.k.a. relative attention), we take an approach similar to the optimization in Music Trans-former (Huang et al., 2018) but generalized to allow arbitrary pairwise labels rather than adhering to a relative position pattern.”
MusicLM: Generating Music From Text (2023, background): “Symbolic representations of music (e.g., MIDI) can also be used to drive the generative process as a form of strong conditioning, as demonstrated by Huang et al. (2019); Hawthorne et al. (2019); Engel et al. (2020).”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “image generation [37]), music generation [27], and analysis of biological sequences [40, 34, 28, 21, 20].”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “Here, modeling long-range dependencies is essential for obtaining good representations (Huang et al., 2019).”