Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset

By Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, Hongsheng Li. arXiv.org, 2024

662 citations recorded by the source corpus.

Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e96c86407b3acf09e5d7729bec201824b717b476

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minfeng Zhu
2. Sicong Leng
3. Li Bing
4. Zishan Guo
5. Shuai Bai
6. Kai Dang
7. Yunfei Chu
8. Shaohui Lin

Citations in the corpus

• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “[245]: MATH-Vision is a high-quality dataset of 3,040 visually contextualized math problems sourced from real competitions.”
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, neutral): “Most of the above studies focus mainly on improving performance in multimodal mathematics tasks [36, 48, 58].”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “…[11], MUIRBENCH [53], ZeroBench(val) [39], HallusionBench [15], GeoBench [2]; • STEM : MMMU(val) [67], MMMU Pro [68], MathVista [32], MathVision [55], MathVerse [70], DynaMath [74], LogicVista [61], WeMath [37], AI2D [26]; • OCR, Chart & Document : OCRBench [31], ChartQAPro [35], ChartMuseum…”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “MathVista [80], MathVision [119], MathVerse [146], DynaMath [155], and WeMath [99] are mathematics benchmarks.”
• Qwen3-Omni Technical Report (2025, positive): “For the specialized domain of mathematical and STEM reasoning, we utilize MathVista (Lu et al., 2024), MathVision (Wang et al., 2024a), MMMU (Yue et al., 2023), and MMMU-Pro (Yue et al., 2024).”
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “During GRPO process, we mix the math datasets of We-Math [39], MathVision [40], Polymath [41], SceMQA [42], Geometry3K [43] as our RL training data.”
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, neutral): “For example, GPT-o1 achieves 73.9% on MathVista [Lu et al., 2023] and 57.0% on MathVerse [Wang et al., 2024a], which is slightly worse than Qwen2.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “…[41], AI2D [55], OCRBench [72], MMVet [168], MME-RealWorld (en) [178], MVBench [63], VideoMME [35], MMMU [170], MathVista [76], MathVision [134], MathVerse [175], DynaMath [189], WeMath [100], Log-icVista [153], MATH500 [45], AIME24 [84], AIME25 [85], GPQA [106], MMLU-Pro [146], GAOKAO…”
• Qwen2.5-Omni Technical Report (2025, positive): “Specifically, we utilize MMMU (Yue et al., 2023) and MMMU-Pro (Yue et al., 2024) for college-level problems evaluation, MathVista (Lu et al., 2024b) and MathVision (Wang et al., 2024b) for math.”
• R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, positive): “Finally, extensive experiments on several benchmarks, including MathVision [25], MathVerse [38], MathVista [17], WeMath [22] and our R1-Onevision-Bench, demonstrate the superior multimodal reasoning performance of our R1-Onevision model.”
• R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, positive): “For evaluation, we adopt 8 widely-used multimodal benchmarks for comprehensively evaluating our proposed StepGRPO, including MathVista [21], MM-Star [5], Math-Vision [36], ChartQA [24], DynaMath [51], HallusionBench [11], MathVerse [48] and MME [10].”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “VideoLLaMA3’s mathematical reasoning capabilities are evaluated through the MathVista [115] and MathVision [116] benchmarks.”
• Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (2024, unknown): “Traditional VQA datasets typically measure a model’s comprehension of object attributes [15, 19], spatial relationships [15], as well as its understanding of documents [27, 28], charts [26], mathematics [23, 40, 46], and open knowledges [25, 35].”