MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

By Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Qinghong Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang. 2023

1,322 citations recorded by the source corpus.

We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, methodology): “Given the significant differences in the model architecture and training data between them, the root cause of the disparity in their capabilities remains elusive, despite conjectures [36, 54]: the amount of training data, the usage of resamplers like Qformer [31], etc .”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “OCRBench [59](with normalized results), MMVet [98], POPE [51], VQAv2 [27], GQA [34], TextVQA [78], ChartQA [61], AI2D [36], RealWorldQA [91], MMMU [99], and MMbench [58].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include VQAv2 [30], GQA [33], POPE [46], MME [24], SEED [40], MMB [52], MM-Vet [87], and MMMU [88].”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “1 (en) [71], MMStar [11], BLINK [36], HallusionBench [41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25…”
Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, neutral): “Instruction :Capture a playful child pre-tending to talk on the phone.”
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, neutral): “[13]; GQA [17]; VisWiz [14]; SQA I : ScienceQA-IMG [32]; VQA T : TextVQA [41]; POPE [28]; MMB: MMBench [31]; LLaVA W : LLaVA-Bench (In-the-Wild) [30]; MM-Vet [50].”
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, methodology): “API models particularly excel on two benchmarks: MMMU [51] and MMVet [50].”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “General multimodal benchmarks encompass: MME [27], RealWorldQA [120], AI2D test [39], MMMU val [130], MMBench-EN/CN test [62], CCBench dev [62], MMVet [128], SEED Image [45], and HallusionBench [30].”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “MM-Vet [86] includes six sub-features from the previous datasets including recognition-focused questions, OCR, and math, providing a diverse while discrete evaluation set.”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “With MLLMs displaying impressive performance on still-images across perception, reasoning, and multi-disciplinary tasks [39, 52, 98, 99], there is increasing interest in evaluating MLLMs’ video understanding capabilities [23, 24, 43, 44, 47, 51, 54, 55, 63, 85, 96].”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “Recent works (Bitton et al., 2023; Yu et al., 2023) evaluate instruction-following and reasoning capabilities, underscoring the growing importance of generative foundation models in practical applications.”
A Survey on Evaluation of Large Language Models (2023, neutral): “This study emphasizes the importance of [21] Social knowledge Specific downstream task Social language understanding MME [43] Multimodal LLMs Multi-modal task Ability of perception and cognition Xiezhi [55] Comprehensive domain knowledge General language task Overall performance across multiple benc”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, neutral): “In response to this need, a number of benchmarks have been launched [17, 45, 49, 90, 97], by evaluating MLLMs with Question Answering (QA) formulation of various perception tasks.”
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “To verify that adding the visual search ability does not impede the general multimodal ability, we evaluate our model on several multimodal benchmarks including MME[10], POPE [24], MMBench [30], SEED-Bench [21], MM-Vet [57], and LLaVA-Bench W [28].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “MMVet [283]: MMVet is a benchmark designed to assess the integrated capabilities of MLLMs on complex tasks.”
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, background): “As the foundation of visual GUI agents, GUI grounding has not received extensive attention in current LVLMs evaluations (Liu et al., 2023b; Yu et al., 2023).”
CogVLM: Visual Expert for Pretrained Language Models (2023, neutral): “We conduct multidimensional evaluations of the models on datasets including MM-Vet [Yu et al., 2023], MMBench [Liu et al., 2023g], SEED-Bench [Li et al., 2023a], LLaVA-Bench [Liu et al., 2023c], POPE [Li et al., 2023c], MMMU [Yue et al., 2023] and MathVista [Lu et al., 2023].”