LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

By Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou and 3 more. Annual Meeting of the Association for Computational Linguistics, 2023

1,466 citations recorded by the source corpus.

Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability. The code and datasets are available at https://github.com/THUDM/LongBench.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b31a5884a8ebe96b6300839b28608b97f8f8ef76

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. A. Gholami
5. Kurt Keutzer
6. Sehoon Kim
7. Shengding Hu
8. Coleman Hooper

Citations in the corpus

• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, positive): “We also evaluate NSA on LongBench (Bai et al., 2023) against state-of-the-art sparse attention meth-Generation ods and Full Attention baseline.”
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, positive): “We apply middle truncation as described in Bai et al. (2024b) for sequences exceeding the model’s context window length.”
• ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, background): “L-Eval (An et al., 2023) and Long-Bench (Bai et al., 2023) are prominent examples, aggregating pre-existing tasks(Kociský et al., 2017; Dasigi et al., 2021; Yang et al., 2018; Huang et al., 2021; Joshi et al., 2017) 2023).”
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “5 in LongBench, which can be found in Table 8, Table 9 and Table 10, respectively.”
• SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “We assess model performances using the LongBench [17] dataset.”
• Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, positive): “Long context understanding As demonstrated in Table 5, we evaluated the models’ performance on LongBench (Bai et al., 2023).”
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, positive): “…we use post-RoPE quantization, both since this is required from an efficiency perspective without a dedicated kernel implementation,and becauseitprovidesbetteraccuracywhenquan-tizing Keys per-token as shown in Appendix L. Table 2 shows the results for LLaMA models for the Wikitext-2 dataset.”
• RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, background): “The synthetic input in R ULER reduces reliance on parametric knowledge, which interferes with the utilization of long-context input in realistic tasks (Shaham et al., 2023; Bai et al., 2023).”
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, methodology): “An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024).”
• A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “For different applications, the focus of memory varies, as it inherently serves the downstream tasks.”
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, methodology): “We conducted comprehensive experiments on LongBench [2] using 17 datasets across various tasks and domains with two backbone models (LLaMa-3-8B and Mistral-7B [18]).”
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “(5) We evaluate LongLLMLingua across five benchmarks, i.e. , Nat-uralQuestions (Liu et al., 2024), LongBench (Bai et al., 2023), ZeroSCROLLS (Shaham et al., 2023), MuSicQue (Trivedi et al., 2022), and LooGLE (Li et al., 2023b), covering a variety of long context scenarios.”
• ASTRA-QA: A Benchmark for Abstract Question Answering over Documents (2026, unknown): “Existing QA benchmarks for RAG include open-domain retrieval, multi-hop reasoning, scientific document QA, and long-context understanding, as seen in ELI5 [26], ASQA [11], HotpotQA [12], Qasper [9], PeerQA [10], and LongBench [13].”
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (2025, unknown): “We also study a more global sense-making setting and construct additional datasets based on LongBench (Bai et al., 2023); see D.4 for details.”
• ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (2024, unknown): “There are six datasets in total, where QMSum (QM), Qasper (QASP), QuALITY (QLTY) are token from SCROLLS (Shaham et al., 2022) and HotpotQA (HQA) MuSiQue (MSQ), MultiFieldQA-en (MFQA) are token from LongBench (Bai et al., 2023).”
• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs (2024, background): “The prompt is provided in the Appendix 6.1.”
• AutoSurvey: Large Language Models Can Automatically Write Surveys (2024, unknown): “Long-form Text Generation The ability to effectively process and generate long-form text is a critical challenge for large language models (LLMs) due to the need to maintain coherence and logical flow over extended passages of text [28–31].”