YaRN: Efficient Context Window Extension of Large Language Models

By Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole. 2023

587 citations recorded by the source corpus.

Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. Code is available at https://github.com/jquesnelle/yarn

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Yuandong Tian
3. Bo Li
4. Mohit Bansal
5. Quoc V. Le
6. Song Han
7. Yashar Mehdad
8. Madian Khabsa

In Large Language Models

1. Damai Dai
2. Bo Li
3. Song Han
4. Fei Huang
5. Zhihong Shao
6. Jingren Zhou
7. Sergey Edunov
8. Madian Khabsa

In Efficiency & Compression

1. Yuandong Tian
2. Beidi Chen
3. Sharan Narang
4. Guangxuan Xiao
5. Song Han
6. Damai Dai
7. Bo Li
8. Fei Huang

Citations in the corpus

• Qwen3 Technical Report (2025, unknown): “Meanwhile, we introduce YARN (Peng et al., 2023) and Dual Chunk Attention (DCA, An et al., 2024) to achieve a four-fold increase in sequence length capacity during inference.”
• Long Context Transfer from Language to Vision (2024, methodology): “To alleviate that, various RoPE-based [65] extension techniques [13, 7, 61, 57, 20] have been proposed to allow for training-free context extrapolation.”
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “Following DeepSeek-V2, we extend the context length of DeepSeek-Coder-V2 to 128K using Yarn (Peng et al., 2023).”
• A Comprehensive Overview of Large Language Models (2023, unknown): “Pre-training: This stream of MLLMs intends to support different modalities using unified end-to-end models.”
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, background): “, 2022) or interpolating position embeddings (Chen et al., 2023; Peng et al., 2023; Han et al., 2023); (3) Using linear or sparse attention mechanisms (Ding et al.”
• Long-context LLMs Struggle with Long In-context Learning (2024, unknown): “These methods (Hao et al., 2022; Chen et al., 2023a; Peng et al., 2023b; Ratner et al., 2023; Xiao et al., 2024) can unlock lots of complex real-world applications from long-document question-answering, multi-document summarization, long-horizon agent tasks, repo-level code understanding.”
• gpt-oss-120b&gpt-oss-20b Model Card (2025, unknown): “We apply rotary position embeddings [14] and extend the context length of dense layers to 131 , 072 tokens using YaRN [15].”
• Qwen2.5-Coder Technical Report (2024, methodology): “To further leverage the model’s extrapolation potential, we applied the YARN mechanism (Peng et al., 2023), enabling the model to handle sequences up to 131,072 (132K) tokens.”
• Effective Long-Context Scaling of Foundation Models (2023, methodology): “We compare with open-source long-context models available in Huggingface Transformers, namely Focused Transformer (Tworkowski et al., 2023b), YaRN (Peng et al., 2023), Xgen (Nijkamp et al., 2023), MPT (MosaicML, 2023b,a) and Together’s L LAMA 2 fork (Together, 2023).”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, unknown): “Scaling attention to longer context will unlock new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and files in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history…”
• RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, unknown): “…proposed to improve length extrapolation in Transformers (Vaswani et al., 2017), including ALiBi (Press et al., 2022), xPOS (Sun et al., 2023b), and RoPE (Su et al., 2023) variants (Chen et al., 2023; Xiong et al., 2023; Peng et al., 2024; Liu et al., 2024b; Ding et al., 2024; Zhu et al., 2024).”
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, unknown): “This speedup stems from our hardware-aligned al-gorithm design to maximize the efficiency of sparse attention architecture: (1) The Blockwise memory access pattern maximizes Tensor Core utilization, (2) The delicate loop scheduling in the kernel eliminates redundant KV transfers.”
• Kimi K2: Open Agentic Intelligence (2025, unknown): “To extend the context window to 128k, we employed the YaRN method [56].”