Rethinking Positional Encoding in Language Pre-training

By Guolin Ke, Di He, Tie-Yan Liu. 2020

366 citations recorded by the source corpus.

How to explicitly encode positional information into neural networks is important in learning the representation of natural languages, such as BERT. Based on the Transformer architecture, the positional information is simply encoded as embedding vectors, which are used in the input layer, or encoded as a bias term in the self-attention module. In this work, we investigate the problems in the previous formulations and propose a new positional encoding method for BERT called Transformer with Untied Positional Encoding (TUPE). Different from all other works, TUPE only uses the word embedding as input. In the self-attention module, the word contextual correlation and positional correlation are computed separately with different parameterizations and then added together. This design removes the addition over heterogeneous embeddings in the input, which may potentially bring randomness, and gives more expressiveness to characterize the relationship between words/positions by using different projection matrices. Furthermore, TUPE unties the [CLS] symbol from other positions to provide it with a more specific role to capture the global representation of the sentence. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness and efficiency of the proposed method: TUPE outperforms several baselines on almost all tasks by a large margin. In particular, it can achieve a higher score than baselines while only using 30% pre-training computational costs. We release our code at https://github.com/guolinke/TUPE.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “This untied design offers more expressiveness by removing the mixed and noisy correlations [21].”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “These errors often involved numbers that were copied from earlier in the demonstration string being swapped, duplicated, or omitted, suggesting that the use of alternative positional encodings could be a promising approach [20], [41].”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Instead of simply adding the position embeddings, we decoupling the position correlation from token embeddings and patch embeddings [67].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “There are also a few works particularly study how to set and learn the bias terms [38,69].”
Shortformer: Better Language Modeling using Shorter Inputs (2021, methodology): “In parallel with our work, Ke et al. (2020) compute attention coefﬁcients by summing two attention matrices, one based on position-position interactions and the other on content-content interactions.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, methodology): “The ﬁne-tuning protocols use the open-source implementation of TUPE [26].”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “Novel ways of encoding positions of tokens in the transformer architecture were proposed, but they were mostly evaluated on natural language processing tasks, showing small performance gains (Ke et al., 2020; He et al., 2020; Wang et al., 2019; Huang et al., 2020).”
Transformers in Time Series: A Survey (2022, methodology): “Besides the absolute and relative positional encodings, there are methods using hybrid positional encodings that combine them together [Ke et al. , 2021].”
SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers (2021, background): “It is well-known that the self-attention block in transformers enables to capture globally sequential information by the means of positional encoding [26].”
Do Transformers Really Perform Badly for Graph Representation? (2021, methodology): “To distinguish the connection of physical and virtual, inspired by [25], we reset all spatial encodings for b ϕ ([ VNode ] ,v j ) and b ϕ ( v i , [ VNode ]) to a distinct learnable scalar.”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, neutral): “Novel ways of encoding positions of tokens in the transformer architecture were proposed, but they were mostly evaluated on natural language processing tasks, showing small performance gains (Ke et al., 2020; He et al., 2020; Wang et al., 2019; Huang et al., 2020).”
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, background): “A recent study [28] (also focuses on BERT) highlights the negative impact of spurious correlations between word and positional embeddings, and proposes to explicitly disentangle the contribution of positional and content embeddings in the attention mechanism.”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, background): “, 2020) 7 3 3 7 3 7 Untied positional scoring (Ke et al., 2021) 7 3 7 7 3 7 Rotary position embedding (Su et al.”
Large-scale chemical language representations capture molecular structure and properties (2021, background): “More recent work 46–48 showed that use of relative position embeddings between tokens results in improved performance.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, neutral): “Attention for machine translation was introduced in Bahdanau et al. (2016), which was retrospectively understood in Ke et al. (2020) as using recurrent neural nets (RNN) for PE.”
Integrating Tree Path in Transformer for Code Representation (2021, methodology): “In the NLP ﬁeld, several works [Shaw et al., 2018, Ke et al., 2020, Raffel et al., 2019, Dai et al., 2019] propose different ways to bias Transformer towards sequential inputs by incorporating positional information into the self-attention module.”