Effective Long-Context Scaling of Foundation Models

By Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oğuz and 11 more. North American Chapter of the Association for Computational Linguistics, 2023

346 citations recorded by the source corpus.

We present an effective recipe to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. Our models are built through continual pretraining from Llama 2 checkpoints with longer text sequences and on a dataset where long texts are upsampled. We perform extensive evaluation using language modeling, synthetic context probing tasks, and a wide range of downstream benchmarks. Across all evaluations, our models achieve consistent improvements on most regular-context tasks and significant improvements on long-context tasks over Llama 2. Moreover, with a cost-effective instruction tuning procedure that is free of expensive annotation, the presented models can already surpass \texttt{gpt-3.5-turbo-16k}‘s overall performance on long-context benchmarks. Alongside these results, we provide an in-depth analysis on each individual component of our method. We delve into Llama’s position encodings and discuss its key limitation in modeling long data. We examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths – ablation results suggest that having abundant long texts in the pretrain dataset is \textit{not} the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5e0cb1c4b91a7486e1c2b15a44a0be56bd74bdc0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Maosong Sun
3. Zhiyuan Liu
4. Jie Tang
5. Mohit Bansal
6. Xu Han
7. Xiao Liu
8. Ximing Lu

Citations in the corpus

• RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “Alternatively, existing fine-tuning based approaches implement “studying" by either directly “memorizing" (Xiong et al., 2023) the input documents or answering practice questions (Wang et al., 2022) without referencing the documents.”
• Qwen3 Technical Report (2025, positive): “5 (Yang et al., 2024b), we increase the base frequency of RoPE from 10,000 to 1,000,000 using the ABF technique (Xiong et al., 2023).”
• ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, methodology): “We also plan to conduct long-context training [38] to overcome the challenge where long context is needed, e.g. in the bug summarization application.”
• Yi: Open Foundation Models by 01.AI (2024, methodology): “Regarding the model architecture, we use standard implementation of the Transformer architecture with Grouped-Query Attention (GQA) [1], SwiGLU [68] activation, and RoPE with an adjusted base frequency (RoPE ABF) [82].”
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, background): “(3) The inferior performance caused by the extended window size of LLMs (Xiong et al., 2023), and the low density as well as the less sensitive position of the questionrelevant key information in the prompt.”
• Qwen2 Technical Report (2024, methodology): “In conjunction with these enhancements, we modified the base frequency of RoPE from 10,000 to 1,000,000 to optimize performance in long-context scenarios (Xiong et al., 2023).”
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “2 (ABF1000w) and outperform ChatGLM3-6B-128K despite being 2.5 times smaller.”
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, neutral): “…to the thrive of long-context LLMs, whose context length has been extended to 128k tokens or more (Anthropic, 2024; OpenAI, 2024c; Reid et al., 2024; GLM et al., 2024; Dubey et al., 2024) through continual training (Xiong et al., 2024; Fu et al., 2024; Bai et al., 2024a; Gao et al., 2024).”
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, background): “Considering the great success of long-context training in LLMs (AI@Meta, 2024; Bai et al., 2024; Xiong et al., 2023) and the efficiency of FlashAttention (Dao et al., 2022), we propose a 3D text-video hybrid attention mechanism.”
• mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024, methodology): “To train the native 8192-context model more efficiently, we adopt a phased training curriculum (Xiong et al., 2024) (Loshchilov and Hutter, 2018) with weight decay 1 e − 5 .”
• Long-context LLMs Struggle with Long In-context Learning (2024, positive): “Recently, different approaches (Xiong et al., 2023; Fu et al., 2024; Liu et al., 2024) help the model to extrapolate to 128K window size with continued pre-training.”
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024, background): “These expansions were achieved not only through context extension—position encoding extension [31; 5] and continual training [47] on long text—but also long context alignment, enabling GLM-4 to effectively handle very long contexts (Cf [1] for technical details).”
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, background): “Recent research has significantly increased the context window sizes for LLMs (Su et al., 2021; Tworkowski et al., 2023; Xiong et al., 2023; Chen et al., 2023b), ensuring that U RIAL with only 1k tokens does not hinder any long-context applications of LLMs.”
• ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “Since Llama3 is pretrained with a much higher RoPE base frequency of 500,000 compared to Llama2, we increased the RoPE base frequency to 150M accordingly.”
• LongEmbed: Extending Embedding Models for Long Context Retrieval (2024, background): “Meanwhile, there have been many successes in extending context window of existing LLMs in a plug-and-play way or via efficient fine-tuning, pushing their context from 4k to 128k (Xiong et al., 2023) and even 2 million tokens (Ding et al., 2024).”
• Agentic Reinforcement Learning for Real-World Code Repair (2025, unknown): “We used a context window of 131K tokens via RoPE scaling (Su et al., 2021; Xiong et al., 2023) for both SFT and RL.”