MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

By Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Z. Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng and 3 more. 2023

1,670 citations recorded by the source corpus.

Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fairly compare MLLMs, instead of struggling in prompt engineering. Besides, with such an instruction, we can also easily carry out quantitative statistics. A total of 30 advanced MLLMs are comprehensively evaluated on our MME, which not only suggests that existing MLLMs still have a large room for improvement, but also reveals the potential directions for the subsequent model optimization. The data are released at the project page https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.

Subfields: Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Lijuan Wang
2. Yu Qiao
3. Linjie Li
4. Hongsheng Li
5. Saining Xie
6. Peng Gao
7. Jianfeng Wang
8. Chunyuan Li

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Maosong Sun
3. Jaehong Yoon
4. Bo Li
5. Limin Wang
6. Yu Qiao
7. Kaipeng Zhang
8. Lijuan Wang

Citations in the corpus

• Improved Baselines with Visual Instruction Tuning (2023, methodology): “We first include We choose to conduct experiments on GQA [20], MME [16], and MM-Vet [54] to examine the representative capabilities of VQA with short answers, VQA with output formatting, and natural visual conversations, respectively.”
• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “In addition to instruction-following models, our zero-initialized attention can be generalized to other vision and language models for parameter-efficient fine-tuning.”
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, methodology): “The MME Benchmark [15] evaluates LMMs’ perception and cognition capabilities through a series of carefully crafted questions across 14 sub-tasks.”
• Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, methodology): “To address these challenges, research communities have introduced a series of benchmarks, including MME [15], MMBench [34], MM-Vet [54], SEED-Bench [21], GAVIE [29], and LAMM-Bench [14].”
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “It is important to note that all the reported figures for the baseline methods are obtained from the MME benchmark (Fu et al., 2023).”
• Emerging Properties in Unified Multimodal Pretraining (2025, positive): “We adopt six widely used benchmarks—MME [20], MMBench ( 1 .”
• Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, neutral): “Instruction :Capture a playful child pre-tending to talk on the phone.”
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “As discussed in the introduction, most existing MLLM benchmarks [17, 49] focus on spatial understanding with systematical definitions of static image tasks.”
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “The previous benchmarks and evaluations have shown that Multimodal LLMs can suffer from hallucination [28, 29], catastrophic forgetting [67] and lack of robustness [11].”
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, positive): “In SEED-X, we adopt a visual tokenizer to unify image comprehension and generation, and pre-train a multi-granularity de-tokenizer to facilitate image generation and high-precision image manipulation in a two-stage manner.”
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “4 F1 score boost on POPE [38] and +18% improvement on MME [16]) across different LVLM families, including LLAVA-1.”
• LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “As for image-based evaluation, we conduct experiments on several widely-adopted benchmarks, including GQA [22], MMB (MMBench) [36], MME [18], POPE [31], SEED [28], SQA I (Image-based setting in ScienceQA) [37], VQA T (TextVQA) [50], VizWiz [20], and VQA v2 (VQA V2) [19].”
• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “We test our model on recently proposed MLLM benchmarks to comprehensively evaluation of the model’s characteristic such as MME (Fu et al., 2023b), Seedbench (Li et al., 2023c), POPE (Li et al., 2023e), LLaVA-Bench (In-the-Wild) (Liu et al., 2023d), MM-Vet (Yu et al., 2023b), MathVista (Lu et al., 20”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “To probe the true capabilities of the emerging LVLMs, the research community has developed many multi-modal benchmarks encompassing a wide range of evaluation axes [27, 14, 38, 51, 39, 21, 26, 50, 46].”
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include VQAv2 [30], GQA [33], POPE [46], MME [24], SEED [40], MMB [52], MM-Vet [87], and MMMU [88].”
• V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “The existing MLLM benchmarks [10, 21, 30] primarily focus on providing comprehensive evaluations across various task categories, and do not adequately challenge or expose the specific limitations of current paradigms mentioned above.”
• MMBench: Is Your Multi-modal Model an All-around Player? (2023, methodology): “To support this claim, we apply the quality control paradigm to other popular multimodal evaluation benchmarks (like MME [17] and SEEDBench [27]) and try to detect the low-quality samples.”