Improve Transformer Models with Better Relative Position Embeddings
By Zhiheng Huang, Davis Liang, Peng Xu, Bing Xiang. 2020
162 citations recorded by the source corpus.
The transformer model has demonstrated superior results on NLP tasks including machine translation and question answering. In this paper, we argue that the position information is not fully utilized in existing work. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review the absolute position embeddings and existing relative position embedding methods. We then propose new methods to encourage increased interaction between query, key and relative position embeddings in the self-attention mechanism. Our most promising approach is a generalization of the absolute position embedding. Our method results in increased accuracy compared to previous approaches in absolute and relative position embeddings on the SQuAD1.1 dataset. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demonstrate empirically that our relative embedding method can be reasonably generalized to and is robust in the inductive perspective. Finally, we show that our proposed method can be effectively and efficiently adopted as a near drop-in replacement for improving the accuracy of large models with little computational overhead.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Gabriel Synnaeve
- Ronan Collobert
- Qiantong Xu
- Tatiana Likhomanenko
- Michael Auli
- Alexei Baevski
- Kentaro Inui
- Dimitris Papailiopoulos
In Computer Vision and Pattern Recognition
- Gabriel Synnaeve
- Luc Van Gool
- Jianlin Su
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Ronan Collobert
- Dacheng Tao
In Model Architectures
- Ronan Collobert
- Gabriel Synnaeve
- Shashank Rajput
- Jun Suzuki
- Tatiana Likhomanenko
- Dimitris Papailiopoulos
- Qiantong Xu
- Kangwook Lee
Citations in the corpus
- RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, background): “…et al. [2016], Shaw et al. [2018], Huang et al. [2018], Dai et al. [2019], Yang et al. [2019], Raffel et al. [2020], Ke et al. [2020], He et al. [2020], Huang et al. [2020] focuses on relative position encoding, which typically encodes the relative position information into the attention mechanism.”
- CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, background): “A number of approaches were thus proposed to encode relative positions in an explicit form [44, 11, 26, 42, 27], leading to improvements in modeling long sequences.”
- SHAPE: Shifted Absolute Position Embedding for Transformers (2021, background): “RPE, which was originally proposed by Shaw et al. (2018), has many variants in the literature (Dai et al., 2019; Raffel et al., 2020; Huang et al., 2020; Wang et al., 2021; Wu et al., 2021).”
- LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[176] Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang.”
- Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Multiplicative strategies were proposed for both RPE (Huang et al., 2020) and APE (Dai et al., 2019).”
- Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Novel ways of encoding positions of tokens in the transformer architecture were proposed, but they were mostly evaluated on natural language processing tasks, showing small performance gains (Ke et al., 2020; He et al., 2020; Wang et al., 2019; Huang et al., 2020).”