How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation

By Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, Wei-Ying Ma. 2016

159 citations recorded by the source corpus.

Recently a few systems for automatically solving math word problems have reported promising results. However, the datasets used for evaluation have limitations in both scale and diversity. In this paper, we build a large-scale dataset which is more than 9 times the size of previous ones, and contains many more problem types. Problems in the dataset are semi-automatically obtained from community question-answering (CQA) web pages. A ranking SVM model is trained to automatically extract problem answers from the answer text provided by CQA users, which signiﬁcantly reduces human annotation cost. Experiments conducted on the new dataset lead to interesting and surprising results.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “Huang et al. (2016) found that the performance of these AWP systems sharply degrades on larger datasets.”
Training Verifiers to Solve Math Word Problems (2021, background): “Dolphin18K (Huang et al., 2016) is a larger dataset containing 18K problems, but solutions are provided only in the form of equations or ﬁ-nal answers.”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “…2017; Amini et al., 2019; Parisotto et al., 2016; Wang et al., 2018; Zou and Lu, 2019) as well as datasets (Ling et al., 2017; Amini et al., 2019; Huang et al., 2016; Saxton et al., 2019) An interesting approach is proposed in (Rabe et al., 2020), in which authors use self-supervised tasks on…”
Are NLP Models really able to Solve Simple Math Word Problems? (2021, background): “multiple-unknown linear word problems (Huang et al., 2016a), geometry (Sachan and Xing, 2017) and probability (Amini et al.”
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (2019, neutral): “These include Dolphin18K (Huang et al., 2016), Math23K (Wang et al.”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, methodology): “Dolphin18K (Huang et al., 2016) We analyze Transformers on LogiQA (Liu et al., 2020), a task with logical reasoning questions such as “David knows Mr. Zhang’s friend Jack, and Jack knows David’s friend Ms. Lin.”
A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers (2020, background): “Dolphin18K (Huang et al., 2016) Arithmetic/algebraic + domain knowledge problems Equation/answer 18K”
Analysing Mathematical Reasoning Abilities of Neural Models (2019, background): “erest in solving algebraic word problems. These questions tend to be crowd sourced or obtained from exercise books, and existing datasets include Allen Institute for AI (2014); Kushman et al. (2014); Huang et al. (2016); Upadhyay & Chang (2016); Wang et al. (2017); Ling et al. (2017). These rang”
Deep Neural Solver for Math Word Problems (2017, methodology): “An exception is the Dolphin18K dataset (Huang et al., 2016) which contains 18,000+ problems.”
MathDQN: Solving Arithmetic Word Problems via Deep Reinforcement Learning (2018, methodology): “According to a recent experimental study (Huang et al. 2016), their accuracies degrade sharply in a larger and more diverse dataset, named Dolphin, harvested from community question-answering web pages.”
Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems (2016, methodology): “494 al., 2015; Zhou et al., 2015; Huang et al., 2016), which computes whether the solution was correct or not, as this is an easy-to-implement metric.”
Modeling Intra-Relation in Math Word Problems with Different Functional Multi-Head Attentions (2019, methodology): “Compared to statistical machine learning-based methods (Kushman et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018; Zhou et al., 2015; Huang et al., 2016) and semantic parsing-based methods (Shi et al.”
Template-Based Math Word Problem Solvers with Recursive Neural Networks (2019, methodology): “Furthermore, as evaluated by (Huang et al. 2016), these methods exhibit low generality and robustness.”
Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems (2020, methodology): “…meth-ods (Bakman, 2007; Yuhui et al., 2010), statistical machine learning methods (Kushman et al., 2014; Zhou et al., 2015; Mitra and Baral, 2016; Huang et al., 2016; Roy and Roth, 2018),semantic parsing methods (Shi et al., 2015; Koncelkedziorski et al., 2015; Huang et al., 2017), and deep…”
Learning from Explicit and Implicit Supervision Jointly For Algebra Word Problems (2016, background): “Recently, Huang et al. (2016) collected a large number of noisily annotated word problems from online forums.”
Learning by Fixing: Solving Math Word Problems with Weak Supervision (2020, unknown): “…2007; Yu-hui et al. 2010), statistical learning methods (Kushman et al. 2014; Zhou, Dai, and Chen 2015; Mitra and Baral 2016; Roy and Roth 2017; Huang et al. 2016), semantic-parsing meth-ods (Shi et al. 2015; Koncel-Kedziorski et al. 2015; Huang et al. 2017) to recent deep learning methods…”
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks (2021, methodology): “, 2010), statistical machine learning methods (Kushman et al., 2014; Zhou et al., 2015; Roy and Roth, 2015, 2016; Mitra and Baral, 2016; Huang et al., 2016; Roy and Roth, 2018), semantic parsing methods (Shi et al.”
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning (2021, background): “…focus on solving arithmetic and algebraic problems, including traditional machine learning approaches (Kushman et al., 2014; Zhou et al., 2015; Huang et al., 2016) and network-based models (Wang et al., 2017, 2018; Xie and Sun, 2019), while solving geometric problems has been rarely…”