LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

By Haoning Wu, Dongxu Li, Bei Chen, Junnan Li. Neural Information Processing Systems, 2024

654 citations recorded by the source corpus.

Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2f9bcfe03ed3c5827036e7a7e672f952e2d1a382

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “For instance, on benchmarks like MLVU, LongVideoBench, and VideoMME, which primarily use video data from YouTube, this improvement may be due to the inclusion of extensive YouTube data in LLaVA-Video-178K, as illustrated in Fig.”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Due to the inherently temporal characteristics of video modalities, specialized benchmarks have been created to evaluate temporal comprehension, underscoring the current limitations of video MLLMs in understanding video content [3, 15, 32, 41, 57, 62].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “Figure 1 provides a detailed assessment of InternVL3’s performance across a diverse set of benchmarks, including MMMU [141], MathVista [80], AI2D [57], ChartQA [91], DocVQA [93], InfographicVQA [92], HallusionBench [45], OCRBench [76], and LongVideoBench [129].”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “1 (en) [71], MTVQA [119], AI2D [55], OCRBench [72], WildVision [78], MME-RealWorld (en) [178], Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVis”
Video Instruction Tuning With Synthetic Data (2024, positive): “We also utilized LongVideoBench (Wu et al., 2024b), EgoSchema (Mangalam et al., 2024), and MLVU (Zhou et al., 2024) for long video understanding, PerceptionTest (P ˘ atr ˘ aucean et al., 2023) for assessing fine-grained perception skills, and VideoMME (Fu et al., 2024) and NExT-QA (Xiao et al.,…”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “…[40] 6,678 473 Manual MovieChat-1K [34] 13,000 564 Manual MoVQA [49] 21,953 992 Manual Video-MME [9] 2,700 1018 Manual LVBench(Ours) 1,549 4,101 Manual efficient method for adapting image-language pre-trained models to dense video understanding through a novel feature pooling strategy.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “…(LVU) benchmarks: (1) MLVU [131]: diverse long-video understanding tasks for videos ranging from 3 minutes to more than 2 hours, (2) LongVideoBench [132]: video reasoning over the referred context within long video-language interleaved inputs, and (3) LVBench [133]: extreme long video…”