MMBench: Is Your Multi-modal Model an All-around Player?
By Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Conghui He, Ziwei Liu, Kai Chen and 1 more. 2023
2,293 citations recorded by the source corpus.
Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: https://github.com/open-compass/VLMEvalKit.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Luke S. Zettlemoyer
- Sergey Levine
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Yu Qiao
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
In Evaluation & Benchmarks
- Jianfeng Gao
- Mohit Bansal
- Noah A. Smith
- Yejin Choi
- Graham Neubig
- Song Han
- Christopher Clark
- Aniruddha Kembhavi
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Given the significant differences in the model architecture and training data between them, the root cause of the disparity in their capabilities remains elusive, despite conjectures [36, 54]: the amount of training data, the usage of resamplers like Qformer [31], etc .”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “They are also known for their ability to produce high-quality wool, which is used in the production of clothing and other products.”
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, neutral): “[13]; GQA [17]; VisWiz [14]; SQA I : ScienceQA-IMG [32]; VQA T : TextVQA [41]; POPE [28]; MMB: MMBench [31]; LLaVA W : LLaVA-Bench (In-the-Wild) [30]; MM-Vet [50].”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “Comprehensive multimodal evaluation benchmarks, such as MME [68], the MMBench series [156], the MMVet series [283, 284], and MMStar [28], provide valuable and widely adopted frameworks for assessing model performance across a diverse set of multimodal tasks.”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include VQAv2 [30], GQA [33], POPE [46], MME [24], SEED [40], MMB [52], MM-Vet [87], and MMMU [88].”
- OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “We present performance in accuracy on benchmarks including POPE [29], MM-Vet [59], MMBench [34], MathVista [35], and our newly developed MagnifierBench under both the multi-choice protocol and the free-form answering protocol.”
- DoRA: Weight-Decomposed Low-Rank Adaptation (2024, methodology): “The fine-tuned models are then evaluated on seven vision-language benchmarks: VQA v2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), VisWiz (Gurari et al., 2018) SQA (Lu et al., 2022), VQA T (Singh et al., 2019), POPE (Li et al., 2023), and MMBench (Liu et al., 2023c).”
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, positive): “With MLLMs displaying impressive performance on still-images across perception, reasoning, and multi-disciplinary tasks [39, 52, 98, 99], there is increasing interest in evaluating MLLMs’ video understanding capabilities [23, 24, 43, 44, 47, 51, 54, 55, 63, 85, 96].”
- TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “Recent Video LLMs (Su et al., 2023; Li et al., 2023c; Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023d; Maaz et al., 2023; Luo et al., 2023; Li et al., 2023g) primarily follow the End-to-End paradigm, with optional temporal modules to model the temporal information across frames.”
- LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “We also include MMBench-Dev [86] and MMMU-Val [156] for multi-discipline evaluation.”
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “MMBench [49] attempts to match predictions with multiple option formats.”
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “As shown in [37], MMBench series [75], MMVet series [138, 139], and MMStar [13].”
- GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, methodology): “These models achieve amazing image-level multimodal abilities, while several benchmarks such as LVLM-eHub (Xu et al., 2023) and MMBench (Liu et al., 2023c) find that these models still have performance bottlenecks when need to be under specific region reference.”
- Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, positive): “Remarkably, our approach attains leading performance in various settings and even surpasses the well-developed Gemini Pro [5], Qwen-VL-Plus [23], and GPT 4V [4] in the complex MMB [24] and MMU [25] dataset, respectively.”
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “Evaluation Benchmarks include VQA-v2 (Goyal et al., 2017); GQA (Hudson & Manning, 2019); VisWiz (Gurari et al., 2018); SQA I : ScienceQA-IMG (Lu et al., 2022); VQA T : TextVQA (Singh et al., 2019); POPE (Li et al., 2023d); MME (Fu et al., 2023); MMB: MMBench (Liu et al., 2023d); LLaVA W : LLaVA-Benc”
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “1 (en) [71], MTVQA [119], AI2D [55], OCRBench [72], WildVision [78], MME-RealWorld (en) [178], Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43],…”
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “We evaluate the general performance of MMICL on both MME (Fu et al., 2023) and MMBench (Liu et al., 2023d) benchmarks 2 .”
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, methodology): “We also evaluated Interleaved MoF on additional benchmarks such as MM-Bench [32] and GQA [21], finding that Interleaved MoF achieves similar performance on these benchmarks.”
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, positive): “We evaluate SEED-X-I on benchmarks specifically designed for evaluating MLLMs, since recent work [31, 21] point out that traditional VQA benchmarks are not tailored for evaluating MLLMs with open-from output.”