Making Language Models Better Reasoners with Step-Aware Verifier

By Yifei Li, Zeqi Lin, Shizhuo Dylan Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen. Annual Meeting of the Association for Computational Linguistics, 2022

328 citations recorded by the source corpus.

Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DiVeRSe (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DiVeRSe has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DiVeRSe on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% to 83.2%).

View Paper: https://www.semanticscholar.org/paper/e826ac71dad8c4ce36d82fb7add43e3d306bb7e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, background): “(2023) proposed to sample various reasoning paths during inference and then derive the final result by majority voting on the answers or through verifiers (Li et al., 2023).”
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025, unknown): “For instance, generating a Chain of Thought (CoT) [15, 16, 17, 18] and incorporating self-verification [19, 20, 21] prior to the final answer have been shown to improve model performance.”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, methodology): “…al., 2023a; Feng et al., 2024), self-reflection (Shinn et al., 2023; Madaan et al., 2023; Wang and Zhao, 2023; Yao et al., 2023b), improved math reasoning (Yue et al., 2023; Luo et al., 2023) and fine-tuning via bootstrapping models (Zelikman et al., 2022; Lewkowycz et al., 2022; Li et al., 2023b).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Consequently, the ultimate outcome is determined through either the aggregation of answers using majority voting or by leveraging a validation mechanism, as posited by [69].”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “Recently, there has been a growing focus on enhancing the reasoning abilities of Large Language Models (LLMs) (Li et al., 2023b; Yuan et al., 2023a; Luo et al., 2023; Mukherjee et al., 2023), because LLMs still lack reasoning skills (Wang et al., 2023b;d; Zheng et al., 2023; Kaddour et al., 2023;…”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, unknown): “Later on, several other works (Drozdov et al., 2023; Zhou et al., 2023c; Nye et al., 2022; Wang et al., 2022a; 2023a; Li et al., 2023b; Wang et al., 2023d; Yu et al., 2023) (Chen et al., 2022).”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “ORM is usually trained with a cross-entropy loss (Cobbe et al., 2021; Li et al., 2023b): where y s is the golden answer of the solution s , y s = 1 if s is correct, otherwise y s = 0 . r s is the sigmoid score of s assigned by ORM.”
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (2024, unknown): “Other approaches include best-of-n decoding, minimum Bayes risk decoding (Lightman et al., 2024; Li et al., 2023; Khanov et al., 2024; Heineman et al., 2024; Wu et al., 2024), and structured search methods such as Tree-of-Thought, Graph-of-Thought, and Monte Carlo Tree Search (Yao et al., 2024;…”
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (2023, background): “We observe that trajectories with wrong answers are mostly incorrect halfway (Li et al., 2023), and the preceding reasoning is still plausible; in other words, we can obtain valid trajectories by correcting the subsequent portions.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “This approach involves using a diverse set of prompts for each question and incorporates a trained verifier with an awareness of reasoning steps.”
Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration (2026, unknown): “…have introduced various enhancements, such as incorporating more complex reasoning chains (Fu et al. 2023), implementing answer verification mechanisms to screen out implausible results (Li et al. 2023), and adopting self-agreement strategies to improve answer reliability (Lin et al. 2024a).”