Extending Context Window of Large Language Models via Positional Interpolation

By Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian. 2023

813 citations recorded by the source corpus.

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, methodology): “However, instead of downscaling frequencies linearly as Chen et al. (2023b), we change the base period from which they are derived.”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, neutral): “Its format is as follows (three instances, for example), 1 { 2 "type": "text_only", 3 "instances": [ 4 { "text": "SAMPLE_TEXT_1" },”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “To determine the scaling factor for RoPE scaling (Chen et al., 2023), we compute it as the ratio of the maximum input sequence length to the context length of the model.”
Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, positive): “During training, we use a 30% masking rate instead of 15% following (Portes et al., 2023) and we remove the Next Sentence Prediction task (Liu et al., 2019).”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, background): “Add the subsequence x i,j = { x i , x i +1 , ..., x j } to the response y rec .”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “In line with prior research on long context LLMs [10, 12], we employ position interpolation techniques [10] to scale the rotary position encoding [51] from 4K to 64K, thereby enabling the processing of extended sequences.”
Long Context Transfer from Language to Vision (2024, methodology): “To alleviate that, various RoPE-based [65] extension techniques [13, 7, 61, 57, 20] have been proposed to allow for training-free context extrapolation.”
Long-CLIP: Unlocking the Long-Text Capability of CLIP (2024, background): “To increase the context length, many works [3] leverage positional embedding interpolation.”
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024, background): “These expansions were achieved not only through context extension—position encoding extension [31; 5] and continual training [47] on long text—but also long context alignment, enabling GLM-4 to effectively handle very long contexts (Cf [1] for technical details).”
Gemma 3 Technical Report (2025, neutral): “We increase RoPE base frequency from 10k to 1M on global self-attention layers, and keep the frequency of the local layers at 10k.”
Long-context LLMs Struggle with Long In-context Learning (2024, neutral): “These methods (Hao et al., 2022; Chen et al., 2023a; Peng et al., 2023b; Ratner et al., 2023; Xiao et al., 2024) can unlock lots of complex real-world applications from long-document question-answering, multi-document summarization, long-horizon agent tasks, repo-level code understanding.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “The positional encoding RoPE has weak extrapolation properties [22], which means the performance drops obviously given an input length exceeds the pre-defined context length.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “These methods include extending the context window (Press et al., 2022; Chen et al., 2023), utilizing recurrent memory (Dai et al.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Large-scale training with 3D parallelism [97, 216], specialized adaptation [94], long text modeling [28, 224].”
MemGPT: Towards LLMs as Operating Systems (2023, methodology): “We generated the DMR question-answer (QA) pairs using a separate LLM that was instructed to write a question from one user to another that could only be answered correctly using knowledge gained from the past sessions (see Appendix for further details).”
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “Following previous practices (Chen et al., 2023; kaiokendev, 2023), we employed a linear scaling strategy, increasing the scaling factor from 1 to 4 and altering the base frequency from 10000 to 100000.”
Effective Long-Context Scaling of Foundation Models (2023, methodology): “1, we show this simple method outperforms a concurrent approach (Chen et al., 2023) for extending LLAMA’s context length and provide a theoretic explanation of its superiority.”