Scaling Laws of RoPE-based Extrapolation

By Xiaoran Liu, Hang Yan, Shuo Zhang, Chen An, Xipeng Qiu, Dahua Lin. 2023

Subfields: Large Language Models, Model Architectures, Scaling & Learning Theory

Citations in the corpus

• Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “While recent work (Liu et al., 2024; Xiong et al., 2023) suggests using larger RoPE bases, our experiments showed degraded performance on GLUE and XTREME-R benchmarks with larger values.”
• ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “Instead of extending the context window of the base model then applying instruction tuning, GradientAI (2024) directly fine-tunes the Llama-3-Instruct, which uses NTK-aware interpolation (Peng et al., 2023) and the formula from Liu et al. (2023b) to scale up θ .”