GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

By Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, Liang Lin. 2021

324 citations recorded by the source corpus.

Automatic math problem solving has recently attracted increasing attention as a long-standing AI benchmark. In this paper, we focus on solving geometric problems, which requires a comprehensive understanding of textual descriptions, visual diagrams, and theorem knowledge. However, the existing methods were highly dependent on handcraft rules and were merely evaluated on small-scale datasets. Therefore, we propose a Geometric Question Answering dataset GeoQA, containing 4,998 geometric problems with corresponding annotated programs, which illustrate the solving process of the given problems. Compared with another publicly available dataset GeoS, GeoQA is 25 times larger, in which the program annotations can provide a practical testbed for future research on explicit and explainable numerical reasoning. Moreover, we introduce a Neural Geometric Solver (NGS) to address geometric problems by comprehensively parsing multimodal information and generating interpretable programs. We further add multiple self-supervised auxiliary tasks on NGS to enhance cross-modal semantic representation. Extensive experiments on GeoQA validate the effectiveness of our proposed NGS and auxiliary tasks. However, the results are still significantly lower than human performance, which leaves large room for future research. Our benchmark and code are released at https://github.com/chen-judge/GeoQA .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Developmental and Educational Psychology, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Lei Wang
5. Bing Tian Dai
6. Ee-Peng Lim
7. Yan Wang
8. Dongxiang Zhang

In Computer Vision and Pattern Recognition

1. Peng Gao
2. Maosong Sun
3. Renrui Zhang
4. Zhiyuan Liu
5. Xu Han
6. Pan Lu
7. Ziyu Guo
8. Rui Zhang

In Natural Language Processing

1. Lei Wang
2. Zhiyuan Liu
3. Maosong Sun
4. Bing Tian Dai
5. Ee-Peng Lim
6. Xu Han
7. Yan Wang
8. Dongxiang Zhang

In Multimodal Learning

1. Peng Gao
2. Renrui Zhang
3. Pan Lu
4. Maosong Sun
5. Tony Xia
6. Wenhu Chen
7. Yan Wang
8. Zhiyuan Liu

Citations in the corpus

• MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, background): “In addition, weakly supervised learning for MWP solving (Hong et al., 2021; Chatterjee et al., 2021) and supervised learning for geometric problem solving (Lu et al., 2021; Chen et al., 2021) have recently attracted much researchers’ attention.”
• Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks (2021, unknown): “…2017; Liang et al., 2018a), to deep learning meth-ods (Ling et al., 2017; Wang et al., 2017, 2018b; Huang et al., 2018; Wang et al., 2018a; Xie and Sun, 2019; Wang et al., 2019; Zhang et al., 2020a,b; Qin et al., 2020; Shen and Jin, 2020; Wu et al., 2020; Chen et al., 2021; Hong et al., 2021a,b).”
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, methodology): “For assessing multimodal capability, works such as Geome-try3K (Lu et al., 2021), GeoQA (Chen et al., 2021), GeoQA+ (Cao and Xiao, 2022), and Uni-Geo (Chen et al., 2022) have employed multimodal information for tackling geometric problems, integrating natural language descriptions with diagrams.”
• Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “In a related context, He-Yueya et al. (2023) introduced an approach that merges an LLM, Codex (Chen et al., 2021b), capable of progressively formalizing word problems into variables and equations, with an external symbolic solver adept at solving the generated equations.”
• TheoremQA: A Theorem-driven Question Answering dataset (2023, methodology): “…University GMAT/GRE NO MathQA (Amini et al., 2019) Arithm/Algebra University AQuA NO MathQA-Python (Austin et al., 2021) Arithm/Algebra University AQuA NO FinQA (Chen et al., 2021c) Finance University CrowdSource NO TAT-QA (Zhu et al., 2021) Finance University CrowdSource NO on our dataset.”
• A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “…this gap, datasets annotated with intermediate rationales such as logic forms (Tafjord et 2019; Lu et al., 2021a), programs (Amini et al., 2019; Chen et al., 2021c,a; Cao and Xiao, 2022; Chen et al., 2022a), and reasoning graphs (Zhang et al., 2021) have been proposed to train models for…”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “In this section, we offer a detailed comparison between M ATH V ERSE and existing multi-modal mathematical benchmarks, i.e., geometry-specific benchmarks [9, 5, 7, 42, 51], MathVista [41], and MMMU [63], from the following four aspects: The Investigation of Diagram Interpretation Capacity.”