A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

By Shen-Yun Miao, Chao-Chun Liang, Keh-Yih Su. 2020

482 citations recorded by the source corpus.

We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “• ASDiv (Miao et al., 2020): https://github.com/chaochun/nlu-asdiv-dataset .”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “We test mathematical abilities on ASDiv (Miao et al., 2020), SVAMP (Patel et al., 2021) and the MAWPS benchmark (Koncel-Kedziorski et al., 2016).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “, 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
Training Verifiers to Solve Math Word Problems (2021, methodology): “The recently developed ASDiv dataset (Miao et al., 2021), which contains 2.3K math word problems, addresses common ﬂaws in prior datasets by ensuring problems have both high diversity and high quality.”
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, methodology): “Over the last few years, there are a growing number of datasets (Kushman et al., 2014; KoncelKedziorski et al., 2016; Upadhyay and Chang, 2017; Huang et al., 2016; Wang et al., 2017; Qin et al., 2020; Miao et al., 2020; Patel et al., 2021) and deep learning-based methods that have been proposed to s”
Are NLP Models really able to Solve Simple Math Word Problems? (2021, methodology): “We consider the widely used benchmark MAWPS (Koncel-Kedziorski et al., 2016) composed of 2373 MWPs and the arithmetic subset of ASDiv (Miao et al., 2020) called ASDiv-A which has 1218 MWPs mostly up to grade level 4 (MAWPS does not have grade level information).”
PAL: Program-aided Language Models (2022, background): “…tasks: (1) mathematical problems (§4.1) from a wide range of datasets including GSM 8 K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2020), and MAWPS (Koncel-Kedziorski et al., 2016); (2) symbolic reasoning (§4.2) from BIG-Bench Hard (Suzgun et al., 2022) Q: On the…”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “Recently, more challenging datasets (Dua et al., 2019; Saxton et al., 2019; Miao et al., 2020; Amini et al., 2019; Hendrycks et al., 2021; Patel et al., 2021) have been proposed to increase the difficulty, diversity or even adversarial robustness.”
Explanations for CommonsenseQA: New Dataset and Models (2021, background): “…al., 2016; Ling et al., 2017; Joshi et al., 2017; Trivedi et al., 2017; Welbl et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019; Talmor et al., 2019; Miao et al., 2020), or (ii) proposing a model with improved answer accuracy (Amini et al., 2019; Bhar-gav et al., 2020; Chen et al., 2020).”
Teaching Small Language Models to Reason (2022, methodology): “We benchmark the proposed method on the following math word problem datasets: (1) GSM8K (Cobbe et al., 2021), (2) MAWPS (Koncel-Kedziorski et al., 2016) and (3) ASDiv (Miao et al., 2021).”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “…with few-shot/zero-shot settings on CoT benchmark (GSM8K (Cobbe et al., 2021), StrategyQA (Geva et al., 2021), SVAMP (Patel et al., 2021), Asdiv (Miao et al., 2021), CommonsenseQA (Talmor et al., 2019)), BigBench-Hard (BBH) (Suzgun et al., 2022), and MMLU (Hendrycks et al., 2021b) as tested in…”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “Solving math problems with LMs Math word problems have been a popular domain for studying reasoning in LMs (Kushman et al., 2014; Ling et al., 2017; Amini et al., 2019; Miao et al., 2020; Hendrycks et al., 2021; Cobbe et al., 2021).”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “The most prominent MWP datasets are MAWPS [Koncel-Kedziorski et al., 2016] and ASDiv-A [Miao et al., 2020] (both for one-unknown arithmetic problems).”
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (2024, unknown): “Experimental Setup We conduct experiments on three testsets: • ASDIV (Miao et al., 2020): an English math word problem corpus with 2,305 instances, each annotated with its problem type and grade level (1 to 6, indicating difficulty).”
On the Advance of Making Language Models Better Reasoners (2022, background): “…reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al.,…”