MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

By Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao and 1 more. 2024

698 citations recorded by the source corpus.

The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

Subfields: Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Renrui Zhang
4. Pan Lu
5. Bo Li
6. Minfeng Zhu
7. Siyuan Huang
8. Hao Shao

In Evaluation & Benchmarks

1. Bo Li
2. Peng Gao
3. Minfeng Zhu
4. Siyuan Huang
5. Renrui Zhang
6. Yuanhan Zhang
7. Peng Jin
8. Pan Lu

Citations in the corpus

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “…we conduct experiments on a series of benchmarks, including MMMU [141] for multidisciplinary reasoning, MathVista [80], MathVision [119], MathVerse [146] for mathematical reasoning, as well as DynaMath [155], WeMath [99] and LogicVista [131] for complementary evaluation on logical reasoning.”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “Part of results are collected from [54, 8, 290, 245, 299, 80] and the OpenCompass leaderboard [46].”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “…[53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31], ChartQAPro [35], ChartMuseum [45],…”
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, neutral): “For evaluation, we employ a diverse set of challenging multimodal benchmarks: • Math-related reasoning: MathVista [Lu et al., 2023], MathVerse [Zhang et al., 2024], and MathVision [Wang et al., 2024a].”
• R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, positive): “Finally, extensive experiments on several benchmarks, including MathVision [25], MathVerse [38], MathVista [17], WeMath [22] and our R1-Onevision-Bench, demonstrate the superior multimodal reasoning performance of our R1-Onevision model.”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “Specifically, we adopt the perception benchmarks including MME [150], MMBench [86], and MMVet [153], and reasoning benchmarks such as MathVerse [164], MathVista [90], and MMMU [156].”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “As image MLLMs have demonstrated exceptional performance in general perception tasks, benchmarks regarding scientific understanding [31], multi-modal mathematical reasoning [43, 72], and multi-disciplinary [69] capabilities have drawn increasing attention.”
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, neutral): “For evaluating the reasoning capability of our Vision-R1, we choose three widely used multimodal math benchmarks: MM-Math [44], MathVista [45], MathVerse [46].”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “…focus on a specific domain for evaluation, including NLVR2 [50] for daily-life VQA, MMMU [61] for colleague-level problem-solving, MathVerse-mv [65] and SciVerse-mv [13] for mathematical and scientific reasoning, BLINK [10] to challenge LMMs, and Mantis-Eval [19] for multi-image understanding.”
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, neutral): “Most of the above studies focus mainly on improving performance in multimodal mathematics tasks [36, 48, 58].”
• R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “For evaluation, we adopt 8 widely-used multimodal benchmarks for comprehensively evaluating our proposed StepGRPO, including MathVista [21], MM-Star [5], Math-Vision [36], ChartQA [24], DynaMath [51], HallusionBench [11], MathVerse [48] and MME [10].”
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, methodology): “…the ability of SPHINX-MoE more concretely and locate its ability level among many developed MLLMs, we evaluate SPHINX-MoE on some recently curated benchmarks, which are listed below: - MMVP (Tong et al., 2024): A benchmark specially crafted to measure MLLMs’ visual understanding capability.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “…extensive experiments across a series of benchmarks, including MMMU [170] for multidisciplinary reasoning, MathVista [76], MathVision [134], and MathVerse [175] for mathematical reasoning, as well as DynaMath [189], WeMath [100], and LogicVista [153] for complementary logical reasoning assessment.”