MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

By Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun-yue Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao. International Conference on Learning Representations, 2023

1,741 citations recorded by the source corpus.

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.

View Paper: https://www.semanticscholar.org/paper/8946891e94831adc8cddb0d32311cce2445c96d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Bo Li
2. Jianfeng Gao
3. Yoshua Bengio
4. Mohit Bansal
5. Zhe Gan
6. Chunyuan Li
7. Percy Liang
8. Trevor Darrell

Citations in the corpus

• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “…[78], MME-RealWorld (en) [178], Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVista [153]; (3) Text-Centric Tasks : MATH500 [65],…”
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, background): “The implementation details is provided in the supplementary materials.”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “…capabilities of InternVL3, we conduct experiments on a series of benchmarks, including MMMU [141] for multidisciplinary reasoning, MathVista [80], MathVision [119], MathVerse [146] for mathematical reasoning, as well as DynaMath [155], WeMath [99] and LogicVista [131] for complementary…”
• A benchmark of expert-level academic questions to assess AI capabilities (2025, unknown): “The framing of the benchmark as "humanity's last exam" feels a bit strong to me.”
• Qwen3-VL Technical Report (2025, unknown): “We evaluate the Qwen3-VL series on a wide range of multimodal reasoning benchmarks, primarily focusing on STEM-related tasks and visual puzzles, including MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), MathVision (Wang et al., 2024b), MathVision-Wild photo (hereafter MathVision WP ) , MathV”
• Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, unknown): “For instance, the proposed method is on par with Qwen-VL-Plus [23] on the MathVista [70] and MMMU [25] benchmark and even surpasses Gemini Pro and GPT-4V on the widely-adopted MMB [24] benchmark.”
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “Notably, various large multimodal models (LMMs) have been developed (Team, 2023; Liu et al., 2023) and demonstrate proficiency on a variety of tasks (Lu et al., 2022; Yue et al., 2023; Zhang et al., 2024b; Lu et al., 2024), offering the potential for multimodal scientific reasoning.”
• CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “We conduct multidimensional evaluations of the models on datasets including MM-Vet [Yu et al., 2023], MMBench [Liu et al., 2023g], SEED-Bench [Li et al., 2023a], LLaVA-Bench [Liu et al., 2023c], POPE [Li et al., 2023c], MMMU [Yue et al., 2023] and MathVista [Lu et al., 2023].”
• Qwen2.5-Omni Technical Report (2025, unknown): “Specifically, we utilize MMMU (Yue et al., 2023) and MMMU-Pro (Yue et al., 2024) for college-level problems evaluation, MathVista (Lu et al., 2024b) and MathVision (Wang et al., 2024b) for math.”
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, unknown): “…(Hendrycks et al., 2021), a challenging collection of 12.5K math problems from math competitions; TheroemQA (Chen et al., 2023b), focusing on theorem applications on problem solving; and MathVista (Lu et al., 2023a), which evaluates the mathematical reasoning ability of LLMs in visual contexts.”
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, unknown): “While VLMs have demonstrated remarkable skill across a wide range of challenging tasks, such as solving advanced college exam questions (Lu et al., 2023; Yue et al., 2024a;b) and spatial understanding tasks (Yang et al., 2024a;b), they also exhibit limitations in visual perception (Zhai et al.,…”
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, methodology): “We report results in Math-Vista [52], MMMU [91], AI2D [40], MME [27], MM-Bench [51], MMBench-Chinese [51], SEED-Bench (Image) [41], LLaVA-Bench (In-the-Wild) [49], QBench [85], MM-Vet [90], HallusionBench [31], ChartQA [56], and POPE [45].”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “5-78B achieved an accuracy of 72.3% on the MathVista test-mini set [163].”
• Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “0 -EN) [46], MM-Vet [96], MMMU [97], MathVista [49], and MMVP [74].”
• DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, background): “Chart/table understanding datasets: OCRBench (Liu et al., 2023b); Hallucination datasets: POPE (Li et al., 2023b); Scientific problem datasets: ScienceQA (Lu et al., 2022a) and MathVista (Lu et al., 2023).”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “The evaluated benchmarks include MMMU (MMMU-Val [51]), MMB (MMBench-EN-Dev [27]), ScienceQA (ScienceQA-Test [30]), AI2D (AI2D-Test [20]), SEED (SEED-Image [21]), and MathVista (MathVista-Mini [29]).”