Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

By Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang and 10 more. Computer Vision and Pattern Recognition, 2024

1,420 citations recorded by the source corpus.

In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs to process sequential visual data is still insufficiently explored, highlighting the lack of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, and reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models with an average accuracy of 75%, compared to 71.9% for GPT-4o. The results also demonstrate that Video-MME is a universal benchmark that applies to both image and video MLLMs. Further analysis indicates that subtitle and audio information could significantly enhance video understanding. Besides, a decline in MLLM performance is observed as video duration increases for all models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data, shedding light on future MLLM development. Project page: https://video-mme.github.io.

View Paper: https://www.semanticscholar.org/paper/22552dd0e7789f175a302055f06444a12e22b652

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zechun Liu
2. Jiabo Ye
3. Haiyang Xu
4. Xiaoqian Shen
5. Changsheng Zhao
6. Fei Huang
7. Yunyang Xiong
8. Qi Qian

Citations in the corpus

• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “Video-MME [69]: Video-MME is a benchmark for evaluating MLLMs in full-spectrum video analysis.”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “For Video-MME [38], MMBench-Video [35], MLVU [154], and LongVideoBench [129], we test with four different settings: 16, 32, 48, and 64 frames, and report the maximum results.”
• LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “We also utilized LongVideoBench (Wu et al., 2024b) (Fu et al., 2024) and EgoSchema (Mangalam et al., 2024) are consider as zero-shot datasets.”
• Qwen3-VL Technical Report (2025, unknown): “We conduct a comprehensive evaluation across a diverse set of video understanding tasks, encompassing general video understanding (VideoMME (Fu et al., 2024a), MVBench (Li et al., 2024b)), temporal video grounding (Charades-STA (Gao et al., 2017)), video reasoning (VideoMMMU (Hu et al., 2025), MMVU ”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, unknown): “…Some [18], Webquest-QA [56]; • Coding : Design2Code [43], Flame-React-Eval [1]; • Spatial Reco & Reasoning : OminiSpatial [24], CV-Bench [51], ERQA [49], All-Angles Bench [65]; • Video Understanding : VideoMME [10], MMVU [71], VideoMMMU [22], LVBench [58], MotionBench [19], MVBench [28]; Setting.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “For the MC-VQA task, we select EgoSchema (Mangalam et al., 2024), Perception-Test (Patraucean et al., 2024), MV-Bench (Li et al., 2023c), and VideoMME (Fu et al., 2024).”
• Video Instruction Tuning With Synthetic Data (2024, unknown): “For instance, on benchmarks like MLVU, LongVideoBench, and VideoMME, which primarily use video data from YouTube, this improvement may be due to the inclusion of extensive YouTube data in LLaVA-Video-178K, as illustrated in Fig.”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “Popular MLLMs (e.g., LLAVA-Interleave (Li et al., 2024a), InternVL (Chen et al., 2024d)) insert visual features into the sequence of embeddings, which can easily exhaust the language model’s context window, resulting in significant memory and computational overhead.”
• LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, unknown): “By efficiently increasing the number of video frames using LongVILA, the performance on VideoMME (Fu et al., 2024a) and long video captioning tasks consistently improves (Figure 2).”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “We assess VideoLLaMA3’s general video understanding capabilities through established benchmarks: (1) Multi-Choice Video Question Answering (MC-VQA) tasks, including MVBench [26], VideoMME [126], EgoSchema [127], and Perception-Test [128].”
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, unknown): “For multi-image/video vision-language benchmarks, we used one multi-image benchmark (BLINK [FHL + 24]) and one video benchmark (VideoMME [FDL + 24]).”
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “…multilingual proficiency (Ahuja et al., 2023), multidisciplinary understanding (Yue et al., 2024; Hu et al., 2024), perception tests (Yuan et al., 2023), mathematical reasoning (Li et al., 2024; Zhang et al., 2024), and video understanding (Li et al., 2023; Ning et al., 2023; Fu et al., 2024a).”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “…EgoSchema [98] and MLVU [169] focusing on long video understanding, PerceptionTest [115] designed to evaluate the perception skills, VideoMME [29] and NeXTQA [141] containing diverse video domains and durations (from minutes to hours), VideoDetailCaption [87] and Video-ChatGPT [96] for video…”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “When applying uniforming sampling, 144 tokens per frame shows better performance than 64 tokens in an 8k context length on VideoMME (Fu et al., 2024) and MLVU Zhou et al. (2024) while worse on EgoSchema Mangalam et al. (2024).”
• Qwen3-Omni Technical Report (2025, unknown): “To evaluate performance on dynamic visual data, we report results on three long video understanding benchmarks: Video-MME (Fu et al., 2024), LVBench (Wang et al., 2024b), and MLVU (Zhou et al., 2025a).”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, unknown): “While LongVideoBench [40], MovieChat [34], MoVQA [49], and Video-MME [9] utilize longer videos to test models, their average duration is still limited to around 10 minutes.”