Transformer Language Models without Positional Encodings Still Learn Positional Information

By Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, Omer Levy. 2022

193 citations recorded by the source corpus.

Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, background): “Recent work suggests that causal models do not require an explicit encoding of position information (Haviv et al., 2022; Kazemnejad et al., 2023), a hypothesis we did not test in this work as we demonstrated that starting from pretrained Llama 2 models is significantly more efficient than training…”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “We hypothesize that the limitation may partially stem from the model’s learning of its own positional encoding with the use of causal masking, as demonstrated in [11].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[36] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy.”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, unknown): “Recent cutting-edge research in the LLM field (Kazemnejad et al., 2024; Haviv et al., 2022) has also indicated that NoPE may offer better length generalization ability.”
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023, background): “, 2022) and interventions in the transformer computation (Wang et al., 2022; Haviv et al., 2022) to study the inner-workings of LMs.”
Extending Context Window of Large Language Models via Positional Interpolation (2023, background): “A recent line of research aims to train Transformers models on short sequences and inference on longer (Press et al., 2022; Sun et al., 2022; Haviv et al., 2022).”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, methodology): “In the I.I.D evaluation of PEs, we demonstrate similar performance across different PEs, in line with observations of Haviv et al. (2022) and Scao et al. (2022b), which makes the choice of optimal positional encoding challenging.”