MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

By Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang and 4 more. Annual Meeting of the Association for Computational Linguistics, 2024

489 citations recorded by the source corpus.

This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly"see"and"read"simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

View Paper: https://www.semanticscholar.org/paper/881ebd6d94065e5157f2290009b147b87eaf4ec9

Citations in the corpus

• Qwen2.5-Omni Technical Report (2025, unknown): “Specifically, we utilize MMMU (Yue et al., 2023) and MMMU-Pro (Yue et al., 2024) for college-level problems evaluation, MathVista (Lu et al., 2024b) and MathVision (Wang et al., 2024b) for math.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “To assess VideoLLaMA3’s ability to understand and reason about multiple images in conjunction, we evaluate the model on several widely used benchmarks, including MMMU-Pro [117], MMMU [118], and BLINK [119].”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “In the left section of Table 6, we provide a comparison of InternVL 2.5’s performance on multidisciplinary reasoning-related benchmarks, including MMMU [289] and MMMU-Pro [290].”
• Qwen3-VL Technical Report (2025, unknown): “…spanning a diverse range of domains—from mathematics and STEM to visual reasoning and puzzle-solving tasks: MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), MathVision (Wang et al., 2024b), MathVision-Wild photo , MathVista (Lu et al., 2023), We-Math (Qiao et al., 2024), MathVerse…”
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, unknown): “• Multi-discipline understanding and reasoning: MMMU [Yue et al., 2024a], MMMU-Pro [Yue et al., 2024b], and EMMA [Hao et al., 2025].”
• Qwen3-Omni Technical Report (2025, unknown): “5-VL-72B, and attains better results on Math & STEM related tasks like MMMU-Pro overall , MathVista mini , and MATH-Vision full , than other vision language models including GPT4-o and Gemini-2.”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, unknown): “1 [30], MMStar [7], BLINK(val) [11], MUIRBENCH [53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31],…”