Solving math word problems with process- and outcome-based feedback

By Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins. 2022

815 citations recorded by the source corpus.

Recent work has shown that asking language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% $\to$ 12.7% final-answer error and 14.0% $\to$ 3.4% reasoning error among final-answer-correct solutions.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, background): “Third, language models capable of strong mathematical reasoning are upstream of a number of research topics, such as reward modeling (Uesato et al., 2022; Lightman et al., 2023), reinforcement learning for reasoning (Polu et al., 2022; Lample et al., 2022), and algorithmic reasoning (Zhou et al.,…”
Let's Verify Step by Step (2023, methodology): “In work closely related to our own, Uesato et al. (2022) compare the impact of outcome and process supervision in the domain of grade school math.”
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025, neutral): “Process Reward Model (PRM) PRM is a reasonable method to guide the model toward better approaches for solving reasoning tasks (Lightman et al., 2023; Uesato et al., 2022; Wang et al., 2023).”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “[126] conducted experiments involving various fine-tuning techniques, prompting strategies, and re-ranking methods to assess their impact on enhancing the performance of a base LLM.”
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, neutral): “Other fine-tuning methods include reward-based supervision (Uesato et al., 2022), content-specific training (Lewkowycz et al.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “Further work has generalized the results of Zelikman et al. (2022), such as Uesato et al. (2022) which demonstrated additional usefulness to “process-based” supervision where incorrect reasoning traces were filtered, recently V-STaR (Hosseini et al., 2024) that demonstrates that training a verifier…”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “Instead of generating an entire response in one pass, beam search selects the most promising intermediate outputs with process reward model [87] at each step, while discarding less the optimal ones.”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “In addition, a good verification model can also offer invaluable feedback for further improvement of LLMs (Uesato et al., 2022; Wang et al., 2023b; Pan et al., 2023).”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, positive): “Moreover, we show that synthetic model-generated rationales, which can be error-prone, are sufficient to teach GenRM how to use verification CoT to pick out tricky errors on math reasoning tasks (see Figure 2, 11, 12, and Appendix D).”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, methodology): “, 2023) or ensembling / reranking multiple times of inferences (Cobbe et al., 2021; Uesato et al., 2022; Wang et al., 2023; Lightman et al., 2023) to improve models’ reasoning performances.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “To make the tree search algorithm generally applicable, our method leverages a learned LLM-based value function v θ ( s ) conditioned on state s and a learned final-step outcome reward model (ORM) ˆ r θ (Uesato et al., 2022) since most tasks can be formulated as sparse-reward problems.”
Process Reinforcement through Implicit Rewards (2025, positive): “…rewards, which provide feedback at each intermediate step rather than only the whole trajectory, have proven effective in inference-time scaling of large language models (LLMs) on challenging reasoning tasks (Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023; Yuan et al., 2024b).”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “The EI method inspired several related approaches (Uesato et al., 2022; Zelikman et al., 2022).”
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “…that facilitates the generation of high-quality CoT processes, which are post-processed and integrated with the original data to create the final Vision-R1-cold dataset. to progressively develop more complex reasoning processes while effectively guiding MLLMs toward enhanced reasoning capability.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “And GPT-4 overperforms any other methods [88], even some huge models particularly tuned for arithmetic problems [114].”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “…et al., 2020; Nye et al., 2021; Wei et al., 2022; Marasovi´c et al., 2021; Lampinen et al., 2022) and correspondingly, that this performance can be improved with further guidance and tool use (Zhou et al., 2022a; Zelikman et al., 2022; Yao et al., 2022; Uesato et al., 2022; Dua et al., 2022).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, neutral): “Thus, we train an instruction reward model (IRM) and a process-supervised reward model (PRM) (Lightman et al., 2023; Uesato et al., 2022; Wang et al., 2024a; Chen et al., 2024a), the former indicates the quality of the evolved instruction and the latter offers feedback for each reasoning step in…”
Training Language Models to Self-Correct via Reinforcement Learning (2024, neutral): “We also assume access to an oracle reward ̂ r ( y , y ∗ ) , such as an answer checker (Uesato et al., 2022), that evaluates the correctness of response y by comparing it with the oracle response y ∗ .”