Generate & Rank: A Multi-task Framework for Math Word Problems

By Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, Qun Liu. 2021

159 citations recorded by the source corpus.

Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-sequence models to transform problem descriptions to mathematical expressions. However, mathematical expressions are prone to minor mistakes while the generation objective does not explicitly handle such mistakes. To address this limitation, we devise a new ranking task for MWP and propose Generate&Rank, a multi-task framework based on a generative pre-trained language model. By joint training with generation and ranking, the model learns from its own mistakes and is able to distinguish between correct and incorrect expressions. Meanwhile, we perform tree-based disturbance specially designed for MWP and an online update to boost the ranker. We demonstrate the effectiveness of our proposed method on the benchmark and the results show that our method consistently outperforms baselines in all datasets. Particularly, in the classical Math23k, our method is 7% (78.4% $\rightarrow$ 85.4%) higher than the state-of-the-art.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, methodology): “One potential method for improving the quality of decoding could involve generating multiple reasoning paths and scoring each of them with a verifier, though this requires training the verifier (Cobbe et al., 2021; Shen et al., 2021; Thoppilan et al., 2022).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
Training Verifiers to Solve Math Word Problems (2021, methodology): “In concurrent work closely related to our own, Shen et al. (2021a) applied a similar approach to solving math word problems, jointly training a model to both generate and rank solutions.”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “Applying pre-trained language models such as BERT (Devlin et al., 2019) was shown to signiﬁcantly beneﬁt the tree expression generation (Lan et al., 2021; Tan et al., 2021; Liang et al., 2021; Li et al., 2021; Shen et al., 2021).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “, 2021), solve math problems (Cobbe et al., 2021; Shen et al., 2021), and even as valuable priors when applied to diverse tasks from different modalities (Lu et al.”
CodeT: Code Generation with Generated Tests (2022, background): “In the domain of solving math word problems, Cobbe et al. (2021) chose the one with highest rank by a trained veriﬁer, and Shen et al. (2021) proposed to jointly train the generator and ranker through a multi-task framework.”
Let's Verify Step by Step (2023, background): “, 2022) or to perform search via rejection sampling (Nichols et al., 2020; Shen et al., 2021; Cobbe et al., 2021).”
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2022, methodology): “Verifiers: To improve CoT, some works train a verifier using chain-level labels: the verifier takes a reasoning chain produced by the model as input and judges the quality of the chain (Cobbe et al., 2021; Shen et al., 2021; Jhamtani and Clark, 2020; Zelikman et al., 2022).”
Self-Edit: Fault-Aware Code Editor for Code Generation (2023, background): “In the domain of solving math problems, Cobbe et al. (2021) and Shen et al. (2021) chose the one with the highest rank by a trained ranker.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “For example, MWP-BERT (Liang et al., 2022b) uses a backbone of BERT (110M) (Devlin et al., 2019) and RoBERTa (123M) (Liu et al., 2019b) for Math Word Problems.”
Large Language Models are Better Reasoners with Self-Verification (2022, methodology): “Shen et al. (2021) proposed the joint training of answer generation and rank with language model.”
LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, background): “Previous work have shown the effectiveness of learned verifiers for sample filtering in domains such as math QA (Shen et al., 2021; Cobbe et al., 2021) and commonsense QA (Li et al.”
On the Advance of Making Language Models Better Reasoners (2022, methodology): “Sample-then-rank is a widely-used strategy in sequence-to-sequence generation (Shen et al., 2021; Cobbe et al., 2021), i.e., generating i.e., to permutate exemplars in the original prompt.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “…incorporated into reinforcement learning pipelines and employed in rejection sampling to align Large Language Models (LLMs) with human preferences (Shen et al., 2021; Bai et al., 2022; Yuan et al., 2023b; Dong et al., 2023; Song et al., 2023; Touvron et al., 2023b; Rafailov et al., 2024; Meng et…”