Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
By Wang Ling, Dani Yogatama, Chris Dyer, Phil Blunsom. 2017
1,014 citations recorded by the source corpus.
Solving algebraic word problems requires executing a series of arithmetic operations—a program—to obtain a final answer. However, since programs can be arbitrarily complicated, inducing them directly from question-answer pairs is a formidable challenge. To make this task more feasible, we solve these problems by generating answer rationales, sequences of natural language and human-readable mathematical expressions that derive the final answer through a series of small steps. Although rationales do not explicitly specify programs, they provide a scaffolding for their structure via intermediate milestones. To evaluate our approach, we have created a new 100,000-sample dataset of questions, answers and rationales. Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Quoc V. Le
- Hannaneh Hajishirzi
- Yoshua Bengio
In Information Systems
- Jamie Callan
- Lei Wang
- Hannaneh Hajishirzi
- Luyu Gao
- Zhuyun Dai
- Ji-rong Wen
- Wayne Xin Zhao
- Ashish Sabharwal
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Graham Neubig
- Noah A. Smith
In Prompting & Reasoning
- Denny Zhou
- Wenhu Chen
- Hannaneh Hajishirzi
- Peter Clark
- Sean Welleck
- Luke S. Zettlemoyer
- Swaroop Mishra
- Jason Wei
Citations in the corpus
- NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “Wang et al. (2017b) and Ling et al. (2017) proposed sequence to sequence solvers for the AWP problems, which are capable of generating unseen expressions and do not rely on sophisticated manual features.”
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “Many existing algebra word problem datasets also contain similar phenomena to what is in DROP (Koncel-Kedziorski et al., 2015; Ling et al., 2017).”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
- Training Verifiers to Solve Math Word Problems (2021, background): “AQuA-RAT (Ling et al., 2017) contains 100K problems, but this dataset unfortunately suffers from both a high degree of problem templatiza-tion and poor quality control of the natural language solutions.”
- Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, methodology): “Furthermore, as directly predicting answers to math problems often requires multiple steps of reasoning, we show that we can improve BERT’s generalization by exposing it to rationales (Ling et al., 2017; Hendricks et al., 2016; Lei et al., 2016).”
- MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (2019, methodology): “We use this representation language to construct MathQA1, a new large-scale, diverse dataset of 37k English multiple-choice math word problems covering multiple math domain categories by modeling operation programs corresponding to word problems in the AQuA dataset (Ling et al., 2017).”
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “Furthermore, we incorporate GRE and GMAT questions from AQaA-RAT (Ling et al., 2017), which emphasizes algebraic word problems.”
- Scaling Instruction-Finetuned Language Models (2022, methodology): “…reasoning dataset (Ling et al., 2017; Camburu et al., 2018; Cobbe et al., 2021; Nye et al., 2021; Zelikman et al., 2022, inter alia ) or focuses on models of substantially smaller scale (Ling et al., 2017; Camburu et al., 2018; Rajani et al., 2019; Talmor et al., 2020; Zelikman et al., 2022).”
- Learning with Latent Language (2017, methodology): “Natural language annotations have been used in conjunction with training examples to guide the discovery of logical descriptions of concepts (Ling et al., 2017; Srivastava et al., 2017), and used as an auxiliary loss for training (Frome et al., 2013), analogously to the Meta+Joint baseline in this…”
- Programmable Agents (2017, background): “Different approaches to this problem focus on networks that infer programs [3, 10, 22], on networks that execute programs [39, 30, 5] and on networks that jointly do both [14, 17, 38, 29, 13]; although there is much overlap between many of these approaches.”
- Large Language Models Can Self-Improve (2022, neutral): “For example, in the supervised learning setting, a model can be fine-tuned using human-annotated rationales (Zaidan et al., 2007; Ling et al., 2017a; Narang et al., 2020; Camburu et al., 2018; Cobbe et al., 2021; Chung et al., 2022).”
- Large Language Models are Zero-Shot Reasoners (2022, background): “, 2014], (3) MultiArith [Roy and Roth, 2015], (4) AQUARAT [Ling et al., 2017], (5) GSM8K [Cobbe et al.”
- Can language models learn from explanations in context? (2022, background): “…tuning models for computer vision (Hendricks et al., 2016; Schramowski et al., 2020; Majumder et al., 2021; Mu et al., 2020), for mathematical problem solving by program induction (Ling et al., 2017), or relational reasoning and reinforcement learning (Andreas et al., 2018; Lampinen et al., 2021).”
- Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “Wang et al. (2017) proposed the Math23k dataset and presented a sequence-to-sequence (seq2seq) approach to generate the mathematical expression (Chiang and Chen, 2019).”
- Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, background): “…(NLEs) for the ground-truth labels at training time and generate such explanations for their decisions at deployment time (Hendricks et al., 2016; Ling et al., 2017; Park et al., 2018; Camburu et al., 2018; Kim et al., 2018; Rajani et al., 2019; Camburu et al., 2020; Narang et al., 2020; Kumar…”