Diagram Understanding in Geometry Questions

By Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni. 2014

123 citations recorded by the source corpus.

Automatically solving geometry questions is a long-standing AI problem. A geometry question typically includes a textual description accompanied by a diagram. The first step in solving geometry questions is diagram understanding, which consists of identifying visual elements in the diagram, their locations, their geometric properties, and aligning them to corresponding textual descriptions. In this paper, we present a method for diagram understanding that identifies visual elements in a diagram while maximizing agreement between textual and visual data. We show that the method's objective function is submodular; thus we are able to introduce an efficient method for diagram understanding that is close to optimal. To empirically evaluate our method, we compile a new dataset of geometry questions (textual descriptions and diagrams) and compare with baselines that utilize standard vision techniques. Our experimental evaluation shows an F1 boost of more than 17% in identifying visual elements and 25% in aligning visual elements with their textual descriptions.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Datasets based on human standardized tests have also been used in AI for several years (e.g. Clark et al., 2016; Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and as part of the NTCIR QALab (NII, 2017), and for the 2016 Allen AI Science Challenge described in the Introduction (Schoenick…”
Solving General Arithmetic Word Problems (2016, background): “There has also been efforts to solve geometry questions by jointly understanding diagrams and associated text (Seo et al., 2014).”
Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012; Hajis”
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (2016, background): “, in geometry (Seo et al. 2014) and mathematics (Hosseini et al.”
My Computer Is an Honor Student - but How Intelligent Is It? Standardized Tests as a Measure of AI (2016, background): “Such questions clearly cannot be answered by information retrieval, and instead require symbolic processing and alignment of textual and algebraic elements (e.g., Hosseini et al., 2014; Koncel-Kedziorski et al., 2015; Seo et al., 2014; Seo et al., 2015) followed by inference.”
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning (2021, background): “Also, while previous datasets emphasize reasoning based on spatial relationships, humans possess diverse reasoning abilities, including analogy [62], geometry[48, 40], arithmetic[63, 20, 21], and physics [13].”
The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers (2018, methodology): “G-ALINGER [91] is an algorithmic work that addresses the geometry understanding and text understanding simultaneously.”
Solving Probability Problems in Natural Language (2017, background): “While for high school algebraic questions standard solvers can be used, more involved domains such as geometrical reasoning have motivated the development of special purpose solvers [Alvin et al., 2014; Seo et al., 2014].”
VisKE: Visual knowledge extraction and question answering by visual verification of relation phrases (2015, background): “Towards Answering Elementary-level Science Questions: Apart from answering generic questions, our approach can be useful for answering more specific questions such as those related to elementary-level general science [7, 21, 33, 20].”
From Textbooks to Knowledge: A Case Study in Harvesting Axiomatic Knowledge from Textbooks to Solve Geometry Problems (2017, methodology): “For diagram parsing, GEOS uses a publicly available diagram parser for geometry problems (Seo et al., 2014).”
Easy Questions First? A Case Study on Curriculum Learning for Question Answering (2016, background): “Some example standardized tests are reading comprehensions (Richardson et al., 2013), algebra word problems (Kushman et al., 2014), geometry problems (Seo et al., 2014), entrance exams (Fujita et al., 2014; Arai and Mat-suzaki, 2014), etc.”
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning (2021, methodology): “…including traditional machine learning approaches (Kushman et al., 2014; Zhou et al., 2015; Huang et al., 2016) and network-based models (Wang et al., 2017, 2018; Xie and Sun, 2019), while solving geometric problems has been rarely investigated (Seo et al., 2014, 2015a; Sachan et al., 2017).”
Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objects (2016, methodology): “Future work involves application of inferred size information in object detection in images and diagrams (Seo et al. 2014), single image depth estimation, and building commonsense knowledge bases.”
Learning to Solve Geometry Problems from Natural Language Demonstrations in Textbooks (2017, methodology): “For diagram parsing, GEOS uses a publicly available diagram parser for geometry problems (Seo et al., 2014) that provides conﬁdence scores for each literal to be true in the diagram.”
A Diagram is Worth a Dozen Images (2016, methodology): “Recently, Seo et al. [6] proposed a method for understanding diagrams in geometry questions that identiﬁes visual elements in a diagram while maximizing agreement between textual and visual data.”