Solving Geometry Problems: Combining Text and Diagram Interpretation

By Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, Clint Malcolm. 2015

266 citations recorded by the source corpus.

This paper introduces GEOS, the first automated system to solve unaltered SAT geometry questions by combining text understanding and diagram interpretation. We model the problem of understanding geometry questions as submodular optimization, and identify a formal problem description likely to be compatible with both the question text and diagram. GEOS then feeds the description to a geometric solver that attempts to determine the correct answer. In our experiments, GEOS achieves a 49% score on official SAT questions, and a score of 61% on practice questions. 1 Finally, we show that by integrating textual and visual information, GEOS boosts the accuracy of dependency and semantic parsing of the question text.

Subfields: Artificial Intelligence, Information Systems, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

MAWPS: A Math Word Problem Repository (2016, background): “…word problems (Shi et al., 2015), logic puzzle problems (Mitra and Baral, 2015), arithmetic word problems (Hosseini et al., 2014; Roy and Roth, 2015), algebra word problems (Kushman et al., 2014; Zhou et al., 2015; Koncel-Kedziorski et al., 2015), and geometry word problems (Seo et al., 2015).”
Parsing Algebraic Word Problems into Equations (2015, background): “…the math domain have studied number word problems (Shi et al., 2015), logic puzzle problems (Mitra and Baral, 2015), arithmetic word problems (Hosseini et al., 2014; Roy and Roth, 2015), algebra word problems (Kushman et al., 2014; Zhou et al., 2015), and geometry word problems (Seo et al., 2015).”
How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation (2016, background): “Seo et al. (2015) focuses on SAT geometry questions with text and diagram provided.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “However, much of this prior work is limited to pre-determined composition structures (Berant et al., 2013; Seo et al., 2015; Neelakantan et al., 2017; Roy and Roth, 2018).”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, background): “On plane geometry questions, (Seo et al., 2015) demonstrated an approach that achieve a 61% accuracy on SAT practice questions.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…Hateful-Memes [111] MAVIS [300], GeomVerse [107], MetaMath-Rendered [281], MapQA [23], GeoQA+ [20], Geometry3K [164], Mathematics UniGeo [26], GEOS [206], CLEVR-Math [144] ChartQA [181], PlotQA [187], FigureQA [105], LRV-Instruction [148], ArxivQA [132], MMC-Inst [149], TabMWP [166], DVQA [104],…”
The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers (2018, methodology): “GEOS [107] can be considered as the first work to tackle a complete geometric word problem as shown in Fig.”
Learning Fine-Grained Expressions to Solve Math Word Problems (2017, background): “Seo et al. (2015) solves a set of SAT geometry questions with text and diagram provided.”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning (2021, background): “Some more works have proposed datasets of middle school math or science problems in more practical and complex scenarios [46, 27, 43, 44, 37].”
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning (2021, methodology): “Compared with the existing dataset GeoS and GeoS++ (Seo et al., 2015a; Sachan et al., 2017), our GeoQA is larger, more diverse, provides additional program annotation, thus serves as a promising benchmark to improve both generalization and interpretability of the multimodal numerical reasoning…”
Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples (2018, methodology): “We took the publicly available training data from (Seo et al., 2015), split the data into sentences, and then annotated each sentence as in Figure 1.”
A Diagram is Worth a Dozen Images (2016, background): “The task of question answering is an important area in the NLP community in several sub-domains including machine comprehension (MC) [50,35,16], science questions [20,7], geometry questions [37], algebra word problems [22,19] and open domain QA [52].”
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks (2017, methodology): “We build a new dataset of demonstrative solutions of geometry problems and show that it can be used to improve GEOS (Seo et al., 2015), the state-of-the-art in solving geom-251 etry problems.”
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, unknown): “The results indicate that Vision-R1-Zero, which applies RL training directly without cold-start initialization, struggles to generate sufficiently long and complex CoT reasoning, thereby limiting its ability to handle intricate reasoning tasks.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “In this section, we offer a detailed comparison between M ATH V ERSE and existing multi-modal mathematical benchmarks, i.e., geometry-specific benchmarks [9, 5, 7, 42, 51], MathVista [41], and MMMU [63], from the following four aspects: The Investigation of Diagram Interpretation Capacity.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…of the LMMs beyond natural scenes, such as abstract scenes and shapes) (Antol et al., 2015; Lu et al., 2021b; Ji et al., 2022), geometry diagrams (Seo et al., 2015; Lu et al., 2021a; Chen et al., 2022a; Cao & Xiao, 2022), figures and charts (Methani et al., 2020; Masry et al., 2022; Kahou et…”
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning (2021, methodology): “The diagram parser first applies Hough Transformation (Shapiro and Stockman, 2001) to extract geometry primitives (points, lines, arcs, and circles), following (Seo et al., 2015).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “Some early datasets are proposed to facilitate research in this domain (Seo et al., 2015; Alvin et al., 2017; Sachan et al., 2017; Sachan and Xing, 2017).”
Query-Reduction Networks for Question Answering (2016, background): “an be comfortably computed in most modern GPUs. 4 Related Work Previous solutions to question answering use open information extraction or textual semantic parsing to query or create a knowledge base [3, 19, 9, 2, 6, 17]. Here, we primarily describe the most related approaches, which use deep neural”