MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms

By Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, Hannaneh Hajishirzi. 2019

894 citations recorded by the source corpus.

We introduce a large-scale dataset of math word problems and an interpretable neural math problem solver by learning to map problems to their operation programs. Due to annotation challenges, current datasets in this domain have been either relatively small in scale or did not offer precise operational annotations over diverse problem types. We introduce a new representation language to model operation programs corresponding to each math problem that aim to improve both the performance and the interpretability of the learned models. Using this representation language, we significantly enhance the AQUA-RAT dataset with fully-specified operational programs. We additionally introduce a neural sequence-to-program model with automatic problem categorization. Our experiments show improvements over competitive baselines in our dataset as well as the AQUA-RAT dataset. The results are still lower than human performance indicating that the dataset poses new challenges for future research. Our dataset is available at https://math-qa.github.io/math-QA/

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “Their work is a remarkable contrast to the literature using formal languages to reason (Roy et al., 2015; Chiang and Chen, 2019; Amini et al., 2019; Chen et al., 2019).”
Training Verifiers to Solve Math Word Problems (2021, methodology): “More recent work has improved performance by designing specialized encoder-decoder architectures (Amini et al., 2019; Chiang and Chen, 2018; Xie and Sun, 2019; Chen et al., 2020; Li et al., 2020), with the strongest results often relying on large pretrained encoders from the BERT family (Chen et…”
Program Synthesis with Large Language Models (2021, methodology): “The second is derived from the MathQA dataset [Amini et al., 2019] but casts the problem solutions as short Python programs.”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Although not ideal, and the sensitive relationship is still present AQuA-RAT (Ling et al., 2017) 36.4% MathQA (Amini et al., 2019) 37.9% Rationales.”
Are NLP Models really able to Solve Simple Math Word Problems? (2021, background): “Recently, researchers have started focusing on solving such MWPs, e.g. multiple-unknown linear word problems (Huang et al., 2016a), geometry (Sachan and Xing, 2017) and probability (Amini et al., 2019), believing that existing work can handle one-unknown arithmetic MWPs well (Qin et al., 2020).”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, background): “Amini et al. (2019); Ling et al. (2017) introduce plug-and-chug multiple choice mathematics problems and focus on sequence-to-program generation.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “…✓ ANLI (Nie et al., 2019) Natural language inference 33.3 ✓ LogiQA (Liu et al., 2021) Multiple-choice question answering 25.0 ✓ HeadQA (Vilares & G´omez-Rodr´ıguez, 2019) Multiple-choice question answering 20.0 ✓ MathQA (Amini et al., 2019) Multiple-choice question answering 20.0 ✓ PROST…”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “, 2017), MathQA (Amini et al., 2019), and SVAMP (Patel et al.”
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers (2020, methodology): “…& Chang, 2015; Wang et al., 2017), which focus on algebraic MWPs; and (4) the Mixed-type MWP corpora (Huang et al., 2016, Ling et al., 2017, Amini et al., 2019), which are large-scale collections of either daily algebra or GRE/GMAT examination MWPs. Table 2 is a comparison of existing…”
Injecting Numerical Reasoning Skills into Language Models (2020, background): “A related line of work has been analyzing the mathematical reasoning abilities of neural models over text (Wallace et al., 2019; Rozen et al., 2019; Ravichander et al., 2019), and on arithmetic problems (Saxton et al., 2019; Amini et al., 2019; Lample and Charton, 2020).”
PAL: Program-aided Language Models (2022, methodology): “Then, the model is expected to apply a similar decomposition to the actual test example, and consecutively reach an accurate final answer (Ling et al., 2017; Amini et al., 2019).”
TALM: Tool Augmented Language Models (2022, background): “The second is MathQA [Amini et al., 2019], selected to measure general reasoning capability rather than knowledge.”
Generative Language Modeling for Automated Theorem Proving (2020, background): “…demonstrated from better pre-training tasks, using various objectives such as auto-regressive generation [19, 20, 21], token masking [22] or sequence masking [36], but resulting language models have so far felt short when applied to reasoning oriented tasks such as algebraic word problems [37, 38].”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…lama-trex qa/closed-book qa Petroni et al. 2019, 2020 liar cls/fact checking Wang 2017 limit other Manotas et al. 2020 math_qa qa/multiple-choice qa Amini et al. 2019 mc_taco qa/binary Zhou et al. 2019 medical_questions_pairs cls/paraphrase McCreery et al. 2020 mocha other/regression Chen et al.…”