MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

By Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo and 2 more. Computer Vision and Pattern Recognition, 2023

1,249 citations recorded by the source corpus.

With the rapid development of Multi-modal Large language Models (MLLMs), a number of diagnostic bench-marks have recently emerged to evaluate the comprehension capabilities of these models. However, most bench-marks predominantly assess spatial understanding in the static image tasks, while overlooking temporal understanding in the dynamic video tasks. To alleviate this issue, we introduce a comprehensive Multi-modal Video understanding Benchmark, namely MVBench, which covers 20 chal-lenging video tasks that cannot be effectively solved with a single frame. Specifically, we first introduce a novel static-to-dynamic method to define these temporal-related tasks. By transforming various static tasks into dynamic ones, we enable the systematic generation of video tasks that require a broad spectrum of temporal skills, ranging from perception to cognition. Then, guided by the task definition, we au-tomatically convert public video annotations into multiple-choice QA to evaluate each task. On one hand, such a distinct paradigm allows us to build MVBench efficiently, without much manual intervention. On the other hand, it guarantees evaluation fairness with ground-truth video an-notations, avoiding the biased scoring of LLMs. More-over, we further develop a robust video MLLM baseline, i.e., VideoChat2, by progressive multi-modal training with di-verse instruction-tuning data. The extensive results on our MVBench reveal that, the existing MLLMs are far from sat-isfactory in temporal understanding, while our VideoChat2 largely surpasses these leading models by over 15% on MVBench. All models and data are available at https://github.com/OpenGVLab/Ask-Anything.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ea3448eb86a233189631d914721e587d45931b64

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “The training data are sourced from the VideoChat2 dataset [18], which embraces data for various video understanding tasks, including 27k for conversation combined from VideoChat [17] and Video-ChatGPT [30], 80k of classification from Kinetics [13] and SthSthV2 [6], 450k caption data from Webvid…”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…Mementos (en) [254], ShareGPT4o-Video (en & zh) [35], Captioning ShareGPT4Video (en & zh) [30], VideoGPT+ (en) [174] VideoChat2-IT (en & zh) [130, 131], EgoTaskQA (en) [99], NTU RGB+D (en) [152], CLEVRER (en) [276], LLaVA-Video (en) [307], FineVideo (en) [67], PerceptionTest (en) [193], HiREST…”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “With MLLMs displaying impressive performance on still-images across perception, reasoning, and multi-disciplinary tasks [39, 52, 98, 99], there is increasing interest in evaluating MLLMs’ video understanding capabilities [23, 24, 43, 44, 47, 51, 54, 55, 63, 85, 96].”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “1 (en) [71], MMStar [11], BLINK [36], HallusionBench [41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25…”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, neutral): “…QA pairs spanning 10 evaluation dimensions from 13 existing data sources, such as MSVD-QA [62], MSRVTT-QA [62], Activitynet-QA [67], etc. MVBench [30] is designed to challenge LVLMs with video tasks that cannot be effectively resolved by single-frame reliance, featuring 4,000 QA pairs derived…”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “To address these limitations, we introduce LVBench, a comprehensive benchmark designed specifically for evaluating MLLMs’ capabilities in long video understanding.”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “For MVBench [65], we conduct testing using 16 frames.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, positive): “In contrast, the LongVILA model (right) demonstrated enhanced performance across a range of frame counts and depths.”
Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “Our experiments show that Video-R1 achieves consistent and significant improvements across a suite of challenging video reasoning benchmarks, including VSI-Bench [24], VideoMMMU [8], MMVU [32], MVBench [13], TempCompass [18], and VideoMME [4].”
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “• For video evaluation , we utilize the existing NExT-QA [57], MVBench [30], Video Detailed Description (VDD) [66], and ActivityNet-QA (Act) [59].”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “It is worth noting that VideoChat2 is trained on more than 700K video instruction-tuning data, while SPHINX-Plus relies solely on image data.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “We employ a updated training version of MVBench [Li et al., 2023d].”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “We evaluate our model on EgoSchema (Mangalam et al., 2024), MVBench (Li et al., 2024b), VideoMME (Fu et al., 2024) and MLVU (Zhou et al., 2024).”
TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “The frame sampling strategies of Video Table 7 summarizes the evaluation results on all fine-grained temporal aspects.”
VideoChat: chat-centric video understanding (2023, neutral): “Remove any repeating sentences, meaningless characters, not English sentences, and so on.”
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “Most of the research has focused on LMMs for image and text [5,29], but research has also been extended to other modalities, such as audiotext [10], video [7, 25, 40], or point clouds [46, 47].”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, positive): “Finally, we report results on general understanding ability using MVBench (Li et al., 2024c), which has a bias towards single-frame appearance features (Krojer et al., 2024; Cores et al., 2024), and TVBench (Cores et al., 2024), which is proposed in the literature as an alternative for general and t”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “…of mPLUG-Owl3 on video and multi-image benchmarks, as it is capable of processing multiple images with an interleaved format. we include VideoChat2 (Li et al., 2023c), Video-LLaMA2 (Cheng et al., 2024), Video-ChatGPT (Maaz et al., 2023), ShareGPT4Video (Chen et al., 2024c), PLLaVA (Xu et al.,…”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, positive): “Based on the same philosophy, MLLMs have been further developed for video processing using video instruction datasets and specialized video adapters [52; 28; 20; 48; 23; 21].”