RULER: What's the Real Context Size of Your Long-Context Language Models?

By Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Boris Ginsburg. arXiv.org, 2024

1,085 citations recorded by the source corpus.

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the"needle") from long distractor texts (the"haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

View Paper: https://www.semanticscholar.org/paper/ac5824e9ff924a937d9eef379d0b581de2417678

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, unknown): “For 2-bit quantization, we also observe significant improvements relative to the baseline approaches, attaining within 0.5 perplexity of the fp16 baseline.”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “For synthetic tasks, we utilize the Needle-In-A-Haystack Single (NIAH-S) benchmark suite from RULER (Hsieh et al., 2024), which includes three increasingly complex tasks: S-NIAH-1 (passkey retrieval), S-NIAH-2 (numerical needle in haystack), and S-NIAH-3 (word-based needle in haystack).”
Qwen3 Technical Report (2025, unknown): “For evaluating long-context processing capabilities, we report the results on the RULER benchmark (Hsieh et al., 2024) in Table 23.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “However, there are relatively few standardized long-context benchmarks for encoder-only models, and most benchmarks, such as Needle-in-a-haystack (Kamradt, 2023) and RULER (Hsieh et al., 2024) retrieval benchmark comprised of over 200,000 long documents.”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “Recently, the Needle-in-a-Haystack task (Hsieh et al., 2024; Kamradt., 2023) has been used to assess the ability of Large Language Models (LLMs) to retrieve long context information.”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, unknown): “Notably, in dynamic tasks such as KV retrieval, MInference can match or even slightly improve performance compared to full attention.”
Gemma 3 Technical Report (2025, unknown): “AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), VQA v2 (Goyal et al., 2017), BLINK (Fu et al., 2024), OK-VQA (Marino et al., 2019), TallyQA (Acharya et al., 2018), Spa-tialSense VQA (Yang et al., 2019), CountBench VQA (Paiss et al., 2023).”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “Critically, existing long-context understanding benchmarks (Bai et al., 2024b; Zhang et al., 2024d; Hsieh et al., 2024) fail to reflect the long-context LLMs’ deep understanding capabilities across diverse tasks.”
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, background): “To measure such development, most benchmarks focus on text-only inputs [Hsieh et al., 2024, Wang et al., 2024a, gkamradt, 2024], while those for long multimodal context remain lacking.”
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “Additionally, we found that the a default RAG solution (top-5 with chunk size as 1200) perform worse than the top long context solution (e.g. GPT-4-Turbo-2024-04-09, Qwen2-72B-Instruct, Llama3 - ChatQA - 2 - 70B ), which suggests that SOTA long context LLMs can handle 32K tokens well within their co”
SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution (2026, unknown): “…demonstrate significant advancement, they still exhibit several limitations: (1) Less-effective context management during code-base search injects excessive issue-irrelevant information into agent’s context, which in turn degrades localization accuracy (Hsieh et al., 2024; Liu et al., 2024).”