A Length-Extrapolatable Transformer

By Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei. 2022

176 citations recorded by the source corpus.

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.

Subfields: Artificial Intelligence, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Model Architectures

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, neutral): “…is a two-digit number that is randomly sampled based on the overall number of tokens in the prompt: (a) Attention expectations over relative distances between key and value embeddings for different frequency regimes, using the bound derived in (Sun et al., 2023) for embedding dimensionality 1024.”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Moreover, these methods only work when combined with windowed attention, which restricts direct attention to long-range tokens [35].”
Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “Compared with the standard Transformer architecture, we include the following modifications: We use M AGNETO [14], a Transformer variant, as the backbone architecture and X P OS [15] relative position encoding for better long-context modeling.”
Retentive Network: A Successor to Transformer for Large Language Models (2023, methodology): “The most related Transformer variant is Lex Transformer (Sun et al., 2023) which implements xPos as position embeddings.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “The KOSMOS-1 model consists of a Magneto-based Large Language Model (LLM) [109] as a general interface and xPos [139] encoders to encode different modalities.”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, methodology): “…embedding methods were proposed to improve length extrapolation in Transformers (Vaswani et al., 2017), including ALiBi (Press et al., 2022), xPOS (Sun et al., 2023b), and RoPE (Su et al., 2023) variants (Chen et al., 2023; Xiong et al., 2023; Peng et al., 2024; Liu et al., 2024b; Ding et al.,…”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “More recently, several methods (Press et al., 2022; Sun et al., 2022; Chen et al., 2023) have been proposed to enable length extrapolation of Transformers, and have been adopted in the training process of long context LLMs such as ChatGLM2-32k (Zeng et al.”
LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens (2023, methodology): “The backbone architecture is M AG - NETO [WMH + 22] with X P OS [SDP + 22] relative position encoding, except that we replace the standard attention with our dilated attention.”