Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
By Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, Zhifang Sui. 2023
1,036 citations recorded by the source corpus.
In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) \textit{Verification}: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning}: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9\%$\to$84.1\% on GSM8K and 28.6\%$\to$33.0\% on MATH). The accuracy can be further enhanced to 89.1\% and 43.5\% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Large Language Models
In Prompting & Reasoning
Citations in the corpus
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, background): “…could potentially act as a linking bridge between the weak reward model and the weak-to-strong learning algorithms; 3) How to efficiently build high-quality process reward models that can provide fine-grained training signals for the reasoning process (Lightman et al., 2023; Wang et al., 2023b).”
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “Test-Time Scaling has been shown to be an effective method to enhance the reasoning abilities of LLMs and MLLMs [108, 94, 87, 70, 120, 36, 152, 125].”
- Improve Mathematical Reasoning in Language Models by Automated Process Supervision (2024, neutral): “In previous studies (Lightman et al., 2023; Wang et al., 2024a,b), both proprietary models such as GPT-4 (OpenAI, 2023) and open-source models such as Llama2 (Touvron et al., 2023) were explored.”
- The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, neutral): “This analysis reveals that some PRMs’ performance in BoN evaluation is predominantly determined by final answer scores rather than intermediate reasoning steps, indicating a model degradation from process-based to outcome-oriented assessment.”
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, neutral): “In domains without automatic verification tools, existing work shows that using majority voting [60], prompting an LLM [19], or training a model-based verifier [18, 42, 29, 59, 35], to decide on a final answer can improve performance on reasoning tasks relative to taking a single sample.”
- Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (2025, neutral): “Some researchers have taken this further by using the search process itself to improve the underlying reasoning model (Wang et al., 2023; Luo et al., 2024).”
- s1: Simple test-time scaling (2025, neutral): “%] means that, with 95% confidence, the true difference between 59K-full and s1K is between -13% and +20%.”
- Process Reinforcement through Implicit Rewards (2025, neutral): “…rewards, which provide feedback at each intermediate step rather than only the whole trajectory, have proven effective in inference-time scaling of large language models (LLMs) on challenging reasoning tasks (Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023; Yuan et al., 2024b).”
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “However, despite much effort in RL algorithms [23, 24, 90, 109, 133, 152, 171, 183] and verifiers [82, 136, 143 .”
- SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (2025, positive): “For the MATH dataset, following prior studies (Lightman et al., 2023; Wang et al., 2023; Sun et al., 2024), we reserve the MATH500 subset as the test set, uniformly sample an additional 500 problems for validation, and combine the remaining 4,000 test problems with the original 7,500 training…”