Are NLP Models really able to Solve Simple Math Word Problems?

By Arkil Patel, Satwik Bhattamishra, Navin Goyal. 2021

1,284 citations recorded by the source corpus.

The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered “solved” with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We can also see that both chain-of-thought and model scaling help significantly on all tasks, as PaLM would have only achieved SOTA on a single task (SVAMP) without both techniques.”
Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2021), (2) the SVAMP dataset of math word problems with varying structures (Patel et al., 2021), (3) the ASDiv dataset of diverse math word problems (Miao et al.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “…tendency to hallucinate facts (Maynez et al., 2020; Ji et al., 2022), difficulties in understanding low-resource languages (Lin et al., 2021), a lack of mathematical skills to perform precise calculations (Patel et al., 2021) and an unawareness of the progression of time (Dhingra et al., 2022).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes signiﬁcant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
Reasoning Like Program Executors (2022, methodology): “On SVAMP , the encoder-only model is not suitable since the answers are out-of-context.”
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, methodology): “Over the last few years, there are a growing number of datasets (Kushman et al., 2014; KoncelKedziorski et al., 2016; Upadhyay and Chang, 2017; Huang et al., 2016; Wang et al., 2017; Qin et al., 2020; Miao et al., 2020; Patel et al., 2021) and deep learning-based methods that have been proposed to s”
Large Language Models are Zero-Shot Reasoners (2022, background): “Among A through C, the answer is The answer is 11 Last Letters The answer is The answer is 12 Coin Flip The answer (Yes or No) is The answer is Therefore, the answer (arabic numerals) is Therefore, the answer is 2 AddSub Therefore, the answer (arabic numerals) is Therefore, the answer is 3 MultiArit”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “For the other more challenging dataset SVAMP, our model has comparable performance with the baseline on 1 -step questions, but achieves signiﬁcantly better results (+ 14 .”
PAL: Program-aided Language Models (2022, background): “…three broad classes of reasoning tasks: (1) mathematical problems (§4.1) from a wide range of datasets including GSM 8 K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2020), and MAWPS (Koncel-Kedziorski et al., 2016); (2) symbolic reasoning (§4.2) from BIG-Bench Hard…”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Despite this, researchers have been able to estimate word frequencies, topics, and genres of documents (Sharoff, 2020), as well as proportions of toxic text (Gehman et al., 2020) 2021) also showed that NLP math solvers rely on simple heuristics to answer math questions.”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, methodology): “We adopt exact match scores as our evaluation metrics for GSM8K, SVAMP, and MultiArith datasets.”
Teaching Small Language Models to Reason (2022, methodology): “We do not evaluate our method on SVAMP (Patel et al., 2021), as the dataset does not have a training split.”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “equation, which is then filled in with specific numeric values from the question (Patel et al., 2021), altogether bypassing the need for encoding numbers into embeddings.”
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, methodology): “We evaluate the quality of confidence estimates across five types of reasoning tasks: 1) Commonsense Reasoning on two benchmarks, Sports Understanding (SportUND) (Kim, 2021) and StrategyQA (Geva et al., 2021) from BigBench (Ghazal et al., 2013); 2) Arithmetic Reasoning on two math problems, GSM8K (C”
A Comprehensive Overview of Large Language Models (2023, neutral): “PaLM webpages, books, wikipedia, news, articles, source code, social media conversations QA: TriviaQA [217], Natural Questions [294], Web Questions [327], TyDiQA-GoldP [261]; CR: PIQA [216], ARC [228], OpenBookQA [304]; IRC: DROP [234], CoQA [233], QuAC [251], SQuADv2 [237], RACE [225]; NLI: ANLI [2”
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, methodology): “We evaluate the performance of our method on five reasoning datasets, including GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), SVAMP (Patel et al., 2021), MultiArith (Roy and Roth, 2015) and MathQA (Amini et al., 2019).”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “On GSM8k [29], SVAMP [92], and AQuA [71], LLMs, as generalists, have competitive performance with most methods that have task-specific designs.”