What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding
By Yu-An Wang, Yun-Nung Chen. 2020
116 citations recorded by the source corpus.
In recent years, pre-trained Transformers have dominated the majority of NLP benchmark tasks. Many variants of pre-trained Transformers have kept breaking out, and most focus on designing different pre-training objectives or variants of self-attention. Embedding the position information in the self-attention mechanism is also an indispensable factor in Transformers however is often discussed at will. Therefore, this paper carries out an empirical study on position embeddings of mainstream pre-trained Transformers, which mainly focuses on two questions: 1) Do position embeddings really learn the meaning of positions? 2) How do these different learned position embeddings affect Transformers for NLP tasks? This paper focuses on providing a new insight of pre-trained position embeddings through feature-level analysis and empirical experiments on most of iconic NLP tasks. It is believed that our experimental results can guide the future work to choose the suitable positional encoding function for specific tasks given the application property.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Naser Ahmadi
- Ghulam Rasool
- Preslav Nakov
- Ravi Prakash Ramachandran
- Paolo Papotti
- Shamoon Siddiqui
- Mohammed Saeed
- Ian E. Nielsen
In Computer Vision and Pattern Recognition
- Ghulam Rasool
- Preslav Nakov
- Ravi Prakash Ramachandran
- Paolo Papotti
- Shamoon Siddiqui
- Mohammed Saeed
- Ian E. Nielsen
- Aakash Tripathi
In Natural Language Processing
- Preslav Nakov
- Paolo Papotti
- Mohammed Saeed
- Naser Ahmadi
- Neema Kotonya
- Francesca Toni
- Todor Mihaylov
- Yonatan Belinkov
In Model Architectures
- Ghulam Rasool
- Preslav Nakov
- Ravi Prakash Ramachandran
- Shamoon Siddiqui
- Ian E. Nielsen
- Aakash Tripathi
- Sabeen Ahmed
Citations in the corpus
- RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, background): “, 2019), and is better at learning positional embeddings (Wang and Chen, 2020).”
- Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “In Wang & Chen (2020), the best performing models exploit both absolute and relative positions.”
- Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “proposed, including methods that have learnable parameters [79, 66].”