∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
By Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Z. Thai, Shuo Wang, Zhiyuan Liu and 1 more. Volume 1, 2024
377 citations recorded by the source corpus.
Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse domains, presented in both English and Chinese. The tasks in $\infty$Bench are designed to require well understanding of long dependencies in contexts, and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. In our experiments, based on $\infty$Bench, we evaluate the state-of-the-art proprietary and open-source LLMs tailored for processing long contexts. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context. We further present three intriguing analyses regarding the behavior of LLMs processing long context.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/f05e84702562cb693dd68d3d1c88072519a7bd71
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, neutral): “Critically, existing long-context understanding benchmarks (Bai et al., 2024b; Zhang et al., 2024d; Hsieh et al., 2024) fail to reflect the long-context LLMs’ deep understanding capabilities across diverse tasks.”
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Whereas its analogue tasks in the text domain, e.g., NIHS (Needle-In-the-HayStack-Search) and Passkey Retrieval, can be effectively conquered by many of the existing long LLMs [10; 53].”
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, positive): “…experiments are conducted on various Long-context LLMs, including LLaMA-3-8B-1M [Gra24], GLM-4-9B-1M [GZX + 24], and Yi-9B-200K [YCL + 24], across benchmarks with context lengths over 1M tokens, such as InfiniteBench [ZCH + 24], RULER [HSK + 24], Needle In A Haystack [Kam23], and PG-19 [RPJ + 20].”
- RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, neutral): “InfiniteBench (Zhang et al., 2024b) includes tasks with length greater than 100K tokens.”
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, background): “After applying DPO for preference alignment, the model’s score on MTBench (Zheng et al., 2024) increased from 6.89 after SFT to 7.25, surpassing even large models such as Llama2-70B-Chat (see Figure 13).”
- Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (2025, unknown): “We set the chunk size to 1200 with an overlap of 100 tokens for NarrativeQA, DetectiveQA, and ∞ Bench, and to 200 for NoCha, following the typical context length distributions of these datasets.”
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “We evaluate our Llama3 - ChatQA - 2 - 70B and Llama3 - ChatQA - 2 - 8B model on the Needle In A Haystack (NIAH) test (Kamradt, 2023).”