Self-Attention with Relative Position Representations

By Peter Shaw, Jakob Uszkoreit, Ashish Vaswani. 2018

2,875 citations recorded by the source corpus.

Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “While the original Transformer used a sinusoidal position signal or learned position embeddings, it has recently become more common to use relative position embeddings [Shaw et al., 2018; Huang et al., 2018a].”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Moreover, our relative positional encodings is also superior to Shaw et al. (2018) on short sequences.”
Music Transformer: Generating Music with Long-Term Structure (2018, methodology): “As the Transformer model relies solely on positional sinusoids to represent timing information, Shaw et al. (2018) introduced relative position representations to allow attention to be informed by how far two positions are apart in a sequence.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, neutral): “…de,fr } + numrefs.1+smooth.exp+test.wmt14/full+tok.13a+version.1.2.9 En–De En–Fr a. Gehring et al. (2017) 25.2 40.5 b. Vaswani et al. (2017) 28.4 41.0 c. Ahmed et al. (2017) 28.9 41.4 d. Shaw et al. (2018) 29 (Grave et al., 2017), and adaptive inputs (Baevski and Auli, 2019).”
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, background): “3 Relative position embedding The authors of Shaw et al. [2018] applied different settings of Equation (1) as following: fq(xm) := W qxm fk(xn, n) := W k(xn + p̃ k r ) fv(xn, n) := W v(xn + p̃ v r) (5)”
Big Bird: Transformers for Longer Sequences (2020, methodology): “• We use position embedding for itc and relative position encoding [80] for etc.”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “Pre-Norm Transformer-RPR: We found Transformer-RPR (Shaw et al., 2018) which simultaneously incorporating relative position information with sinusoidal position encodings for sequences in pre-norm style could outperform the pre-norm Transformer with the same encoder depth.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “For an input sequence of length N , it requires a space complexity of OpN2dq [19, 20, 21] to store the relative position embedding for each token.”
Reasoning Over Semantic-Level Graph for Fact Checking (2019, background): “However, this is unacceptable in practice because the representation learning procedure will take huge memory space, which is also observed by Shaw et al. (2018).”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Its ability to directly encode long-range interactions and its parallelizability, has led to state-of-the-art performance for various tasks [83,40,26,69,75,25,56].”
Learning Deep Transformer Models for Machine Translation (2019, neutral): “As for our approach, considering the post-norm case first, we can see that our Transformer baselines are superior to Vaswani et al. (2017) in both Base and Big cases.”
Attention Augmented Convolutional Networks (2019, methodology): “As a solution, we propose to extend the use of relative position encodings [37] to two dimensions and present a memory efficient implementation based on the Music Transformer [18].”
Towards Robust Vision Transformer (2021, background): “Concerning the ways of positional encoding, learned absolute, sin-cos absolute, learned relative [35], input-conditioned [3] position representations are compared.”
Imitating Interactive Intelligence (2020, methodology): “Together, the 65 vectors comprised the input to an 8-layer, 8-head transformer Vaswani et al. (2017) with size 512 embeddings and MLP layers, using relative position encoding Shaw et al. (2018).”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, background): “…crossmodal attention block adapts directly from the low-level feature sequence (i.e., Z [0] β in Figure 3(b)) and does not rely on self-attention, which makes it different from the NMT encoder-decoder architecture (Vaswani et al., 2017; Shaw et al., 2018) (i.e., taking intermediate-level features).”