MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
By Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun and 12 more. 2023
2,391 citations recorded by the source corpus.
We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Percy Liang
- Yejin Choi
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Yu Qiao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Alan L. Yuille
In Multimodal Learning
In Evaluation & Benchmarks
Citations in the corpus
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “We pro-pose Multimodal-CoT that incorporates language and vision modalities into a two-stage framework that separates rationale generation and answer inference, so answer inference can leverage better generated rationales from multimodal information.”
- InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, methodology): “We report results in Math-Vista [52], MMMU [91], AI2D [40], MME [27], MM-Bench [51], MMBench-Chinese [51], SEED-Bench (Image) [41], LLaVA-Bench (In-the-Wild) [49], QBench [85], MM-Vet [90], HallusionBench [31], ChartQA [56], and POPE [45].”
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “In the first stage, as shown in Fig.”
- Qwen3-Omni Technical Report (2025, neutral): “5-VL-72B, and attains better results on Math & STEM related tasks like MMMU-Pro overall , MathVista mini , and MATH-Vision full , than other vision language models including GPT4-o and Gemini-2.”
- Qwen2.5-Omni Technical Report (2025, positive): “Specifically, we utilize MMMU (Yue et al., 2023) and MMMU-Pro (Yue et al., 2024) for college-level problems evaluation, MathVista (Lu et al., 2024b) and MathVision (Wang et al., 2024b) for math.”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “MMMU [289] and MMMU-Pro [290] are multidisciplinary reasoning benchmarks, while MathVista [163], MATH-Vision [245], MathVerse [299], and OlympiadBench [80] are mathematics benchmarks.”
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, positive): “To our knowledge, MMAU stands as the first comprehensive benchmark to rigorously assess these capabilities, filling a critical gap in the evaluation of LALMs. evaluation (Hendrycks et al., 2020; Yue et al., 2024).”
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “- AesBench (Huang et al., 2024): An expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs. - MMMU (Yue et al., 2023a) & CMMMU (Ge et al., 2024): An English and a Chinese benchmark, respectively, aiming to solve massive multi-discipline tasks, which need…”
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, neutral): “…captioning tasks like Nocaps Agrawal et al. (2019), Flickr30K Plummer et al. (2015), multimple choice tasks like A-OKVQA Schwenk et al. (2022), MMMU Yue et al. (2023), complex embodied reasoning task like PCA-Bench Chen et al. (2024; 2023), tasks requiring detailed OCR ablitily like OCR-VQA Mishra…”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Additionally, the evaluation was mainly supported by VLMEvalKit [23].”
- What matters when building vision-language models? (2024, positive): “The detailed performance of Idefics2 across each category of MMMU (Yue et al., 2024) is present in Table 16.”
- LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, positive): “Comparative analysis shows that InternVL2-40B outperforms GPT-4V and Gemini Pro 1.5 in the MM-Bench [20] and OpenCompass benchmarks [62], while maintaining comparable performance in MMMU [40].”
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, positive): “…VLMs have demonstrated remarkable skill across a wide range of challenging tasks, such as solving advanced college exam questions (Lu et al., 2023; Yue et al., 2024a;b) and spatial understanding tasks (Yang et al., 2024a;b), they also exhibit limitations in visual perception (Zhai et al.,…”
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “We construct 2 new tasks for multi-image mathematical (MathVerse [65]) and scientific (SciVerse [13]) comprehension, and utilize 3 existing benchmarks, Mantis-Eval [19], BLINK [10], and MMMU [60], with 4.1K in total.”
- GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “Their performance on a series of benchmarks (Kazemzadeh et al., 2014; Goyal et al., 2016; Hendrycks et al., 2020; Saikh et al., 2022; Lu et al., 2022; Zhong et al., 2023; Yue et al., 2023) also showcases remarkable capabilities on vision-and-language understanding and reasoning.”
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Along with the development of MLLMs, new benchmarks are continuously created to facilitate comprehensive and in-depth analysis of MLLMs [50; 27; 9; 21].”
- BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “Although there have been prior efforts to incorporate art-related questions into evaluation [87], such attempts primarily focused on questions requiring expert-level knowledge, including deducing an artist’s name and understanding historical contexts, rather than on direct image comparison.”
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, methodology): “The Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark (Yue et al., 2024) is a comprehensive dataset designed to evaluate multimodal AI models on college-level tasks requiring subject-specific knowledge and deliberate reasoning.”
- VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, neutral): “• Multi-discipline understanding and reasoning: MMMU [Yue et al., 2024a], MMMU-Pro [Yue et al., 2024b], and EMMA [Hao et al., 2025].”