Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems

By Shyam Upadhyay, Ming-Wei Chang. 2016

64 citations recorded by the source corpus.

We propose a new evaluation for automatic solvers for algebra word problems, which can identify mistakes that existing evaluations overlook. Our proposal is to evaluate such solvers using derivations, which reflect how an equation system was constructed from the word problem. To accomplish this, we develop an algorithm for checking the equivalence between two derivations, and show how derivation annotations can be semi-automatically added to existing datasets. To make our experiments more comprehensive, we include the derivation annotation for DRAW-1K, a new dataset containing 1000 general algebra word problems. In our experiments, we found that the annotated derivations enable a more accurate evaluation of automatic solvers than previously used metrics. We release derivation annotations for over 2300 algebra word problems for future evaluations.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, background): “Over the last few years, there are a growing number of datasets (Kushman et al., 2014; KoncelKedziorski et al., 2016; Upadhyay and Chang, 2017; Huang et al., 2016; Wang et al., 2017; Qin et al., 2020; Miao et al., 2020; Patel et al., 2021) and deep learning-based methods that have been proposed to s”
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (2019, neutral): “, 2014), and DRAW-1K (Upadhyay and Chang, 2017).”
Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems (2018, background): “…et al., 2014; Hosseini et al., 2014; Roy et al., 2015; Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Roy et al., 2016; Upadhyay et al., 2016; Upadhyay and Chang, 2017; Roy and Roth, 2018; Wang et al., 2018 In terms of how to learn the model for solving math word problems, the prior work can…”
Analysing Mathematical Reasoning Abilities of Neural Models (2019, background): “These questions tend to be crowd sourced or obtained from exercise books, and existing datasets include Allen Institute for AI (2014); Kushman et al. (2014); Huang et al. (2016); Upadhyay & Chang (2016); Wang et al. (2017); Ling et al. (2017).”
Learning from Explicit and Implicit Supervision Jointly For Algebra Word Problems (2016, methodology): “When the explicit supervision is available (i.e., 4Our approach can be easily extended to other structured learning algorithms such as Structured SVM (Taskar et al., 2004; Tsochantaridis et al., 2004).”
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, unknown): “A surge of datasets has been proposed to facilitate the research in this domain (Upadhyay & Chang, 2017; Amini et al., 2019; Miao et al., 2020; Cobbe et al., 2021).”
The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers (2018, unknown): “3) DRAW1K [60].”
Point to the Expression: Solving Algebraic Word Problems Using the Expression-Pointer Transformer Model (2020, unknown): “, 2014)5, DRAW-1K (Upadhyay and Chang, 2016)6, and MAWPS (Koncel-Kedziorski et al.”
A Bottom-Up DAG Structure Extraction Model for Math Word Problems (2021, methodology): “We conduct experiments on two datasets, DRAW1K and Math23K: • DRAW1K (Upadhyay and Chang 2017) contains 1,000 algebra word problems, the questions are like the example shown in Table 1.”
Tree-structured Decoding for Solving Math Word Problems (2019, methodology): “This two-step pipeline model was further extended with tree-based features, ranking style retrieval models and so on (Upadhyay and Chang, 2017; Roy and Roth, 2017).”
Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems (2020, unknown): “, 2014), DRAW-1K (Upadhyay and Chang, 2017), MaWPS (Koncel-Kedziorski et al.”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “…508 E A DD S UB (Hosseini et al., 2014) 395 E Only addition and subtraction M ULTI A RITH (Roy and Roth, 2015) 600 E Multi-step reasoning D RAW -1K (Upadhyay and Chang, 2017) 1K E M ATH 23K (Wang et al., 2017) 23K E Chinese A PE 210K (Zhao et al., 2020) 210K E Chinese K6 (Yang et al., 2023) 600 E…”