An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation

By Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Douglas Eck. 2018

82 citations recorded by the source corpus.

Subfields: Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Jesse Engel
2. Ruslan Salakhutdinov
3. Curtis Hawthorne
4. Ian Simon
5. Louis-philippe Morency
6. Yao-Hung Hubert Tsai
7. Adam Roberts
8. Alec Radford

In Model Architectures

1. Ruslan Salakhutdinov
2. Quoc V. Le
3. Shaojie Bai
4. Zihang Dai
5. Yoshua Bengio
6. Zhilin Yang
7. Ilya Sutskever
8. William W. Cohen

Citations in the corpus

• Generating Long Sequences with Sparse Transformers (2019, methodology): “(Huang et al., 2018) apply Transformers to MIDI generation with an efficient relative attention.”
• Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Previously, the idea of relative positional encodings has been explored in the context of machine translation (Shaw et al., 2018) and music generation (Huang et al., 2018).”
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (2019, background): “However, scaling attention to extremely long sequences is computationally prohibitive since the space complexity of self-attention grows quadratically with sequence length [21].”
• Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (2018, methodology): “, 2017) with relative self-attention, which has previously shown compelling results in generating music with longer-term coherence (Huang et al., 2018).”
• Modeling Self-Repetition in Music Generation using Generative Adversarial Networks (2019, background): “Some prior works focus on modeling long term structures in music generation (Eck & Schmidhuber, 2002; Huang et al., 2018; Roberts et al., 2018).”
• Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “Previously, the idea of relative positional encodings has been explored in the context of machine translation (Shaw et al., 2018) and music generation (Huang et al., 2018).”
• Generative Models for Graph-Based Protein Design (2019, background): “The dense variation of this problem was explored in [34] and [35].”
• Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (2019, methodology): “…allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Lee et al., 2018; Wang et al., 2018; Tsai et al., 2019a).”
• Improve Transformer Models with Better Relative Position Embeddings (2020, background): “This phenomena has been realized and the relative position representation has been proposed in Shaw et al. (2018); Huang et al. (2018), in the context of encoder decoder machine translation and music generation respectively.”