Are We on the Right Way for Evaluating Large Vision-Language Models?

By Lin Chen, Jinsong Li, Xiao-wen Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin and 1 more. Neural Information Processing Systems, 2024

885 citations recorded by the source corpus.

Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. However, we dig into current evaluation works and identify two primary issues: 1) Visual content is unnecessary for many samples. The answers can be directly inferred from the questions and options, or the world knowledge embedded in LLMs. This phenomenon is prevalent across current benchmarks. For instance, GeminiPro achieves 42.9% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks over 24% on average. 2) Unintentional data leakage exists in LLM and LVLM training. LLM and LVLM could still answer some visual-necessary questions without visual content, indicating the memorizing of these samples within large-scale training data. For example, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%. Both problems lead to misjudgments of actual multi-modal gains and potentially misguide the study of LVLM. To this end, we present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs' multi-modal capacities with carefully balanced and purified samples. These samples are first roughly selected from current benchmarks with an automated pipeline, human review is then involved to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities. Moreover, two metrics are developed to measure data leakage and actual performance gain in multi-modal training. We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8a9e11addba791860a9dbf15de75cc28d2cf844c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “MMStar [28]: MMStar is a benchmark for evaluating the multimodal capabilities of MLLMs.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, neutral): “However, many popular MLLMs are primarily trained to respond to instructions based on visual inputs, employing a decoder-only autoregressive design as a single black box.”
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, neutral): “MMStar [6] (data contamination); 3.”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “As shown in [37], MMBench series [75], MMVet series [138, 139], and MMStar [13].”
Qwen3-Omni Technical Report (2025, neutral): “To assess performance in general visual question answering, the model is evaluated on MMStar (Chen et al., 2024a), HallusionBench (Guan et al., 2024), and MM-MT-Bench (Agrawal et al., 2024).”
Qwen3-VL Technical Report (2025, neutral): “A similar upward trend is observed on other benchmarks, such as MMStar, where the score rises from 68.1 (2B, thinking) to 75.3 (8B, thinking).”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “We conduct experiments on several multimodal reasoning benchmarks, including MMStar [10], MMBench [35], MMVet [64], MathVista [37], AI2D [25], and Hal-lusionBench [18], and observed that LLaVA-CoT offers two primary advantages: First, enabling the model to perform structured reasoning independently…”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “However, SpatialVLM relies on language descriptions of objects as input, while LLMs can already resolve some spatial queries even without visual data [20].”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “The vision-language community has recently made significant advancements in the field of large image-language models [36, 35, 37, 10, 13, 11, 58, 59, 2, 72, 33].”
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models (2024, neutral): “However, research in both LLMs [86, 76] and LMMs [9] has indicated that data contamination can significantly skew benchmark scores.”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “As highlighted in MM-star [5], many multimodal benchmarks can be effectively solved using pure text input alone.”
Qwen2.5-Omni Technical Report (2025, neutral): “We assess our model on several representative video understanding tasks like Video-MME (Fu et al., 2024a), MVBench (Li et al., 2024a), and EgoSchema (Mangalam et al., 2023).”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “For evaluation, we adopt 8 widely-used multimodal benchmarks for comprehensively evaluating our proposed StepGRPO, including MathVista [21], MM-Star [5], Math-Vision [36], ChartQA [24], DynaMath [51], HallusionBench [11], MathVerse [48] and MME [10].”
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “1 [30], MMStar [7], BLINK(val) [11], MUIRBENCH [53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31],…”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “Determining whether a benchmark truly needs visual input to be solved has been a persistent challenge in vision-language research [2, 21, 43, 87].”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “1 (en) [71], MMStar [11], BLINK [36], HallusionBench [41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25…”