MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

By Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang and 7 more. 2024

1,918 citations recorded by the source corpus.

In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Rongyao Fang
3. Binyuan Hui
4. Jingren Zhou
5. Sinan Tan
6. Xiaohuan Zhou
7. Shuai Bai
8. Kai Dang

In Natural Language Processing

1. Fei Huang
2. Rongyao Fang
3. Binyuan Hui
4. Jingren Zhou
5. Kai Dang
6. Yichang Zhang
7. Junyang Lin
8. Jinze Bai

In Evaluation & Benchmarks

1. Fei Huang
2. Jingren Zhou
3. Sinan Tan
4. Shuai Bai
5. Kai Dang
6. Junyang Lin
7. Jinze Bai

Citations in the corpus

• Qwen2 Technical Report (2024, methodology): “The datasets for evaluation include MMLU (Hendrycks et al., 2021a) (5-shot), MMLU-Pro (Wang et al., 2024) (5-shot), GPQA (Rein et al., 2023) (5shot), Theorem QA (Chen et al., 2023a) (5-shot), BBH (Suzgun et al., 2023) (3-shot), HellaSwag (Zellers et al., 2019) (10-shot), Winogrande (Sak-aguchi et…”
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, neutral): “5-Air on seven benchmarks, including MMLU-Pro [43], AIME 24, MATH 500 [14], SciCode [36], GPQA [30], Humanity’s Last Exam (HLE) [28], and LiveCodeBench (LCB) [19] 2 .”
• Qwen3-VL Technical Report (2025, positive): “(1) Knowledge: MMLU-Pro (Wang et al., 2024f), MMLU-Redux (Gema et al., 2024), GPQA (Rein et al., 2023), Su-perGPQA (Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wa”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, neutral): “…capabilities (Liu et al., 2023c,b, 2024a; Li et al., 2024b; Dai et al., 2023; Zhu et al., 2023; Zhang et al., 2023; Gao et al., 2023; Ye et al., 2023a,b; Zhao et al., 2023; Li et al., 2023; Monajatipoor et al., 2023; Zhao et al., 2024; Li et al., 2024c; Lin et al., 2024; Zhang et al., 2024).”
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (2025, neutral): “Future work should investigate scaling test time computation through multi-turn interactions for contextual reasoning, value models for trajectory assessment, and multi-agent scenarios for developing more sophisticated reasoning strategies.”
• A benchmark of expert-level academic questions to assess AI capabilities (2025, neutral): “The framing of the benchmark as "humanity's last exam" feels a bit strong to me.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “…[170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25 [85], GPQA [106], MMLU-Pro [146], GAOKAO [177], IFEval [185], SGP-Bench [102], VSI-Bench [161], ERQA [121], SpaCE-10 [38], and OmniSpatial [50]. community.”
• R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, neutral): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. and science [27].”
• Kimi K2: Open Agentic Intelligence (2025, neutral): “Finally, general capabilities are assessed using MMLU [24], MMLU-Redux [18], MMLU-Pro [77], IFEval [91], Multi-Challenge [65], SimpleQA [79], and LiveBench [81] (as of 2024-11-25).”
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, neutral): “By integrating hierarchical token compression with block selection within a trainable architecture, we achieve faster training and inference while matching or surpassing Full Attention performance.”
• Qwen2.5-Omni Technical Report (2025, neutral): “Specifically, we utilize MMLU-Pro (Wang et al., 2024f), MMLU-redux (Gema et al., 2024) and Livebench0803 (White et al., 2024) for general evaluation, GPQA (Rein et al., 2023), GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) for mathematics & science, HumanEval (Chen et al., 2021), MBPP”