Let's Verify Step by Step
By H. Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe. 2023
3,887 citations recorded by the source corpus.
In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.
Subfields: Artificial Intelligence, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Hannaneh Hajishirzi
- Yejin Choi
In Prompting & Reasoning
- Graham Neubig
- Sean Welleck
- Jie Huang
- Michael R Douglas
- Xinyun Chen
- Pengfei Liu
- Zhiyuan Liu
- Swaroop Mishra
Citations in the corpus
- Llemma: An Open Language Model For Mathematics (2023, background): “Although achieving strong mathematical reasoning is an important target, it is difficult to assess the correctness of models’ answers and processes, especially as models become more capable (Bowman et al., 2022; Uesato et al., 2022; Lightman et al., 2023; Cobbe et al., 2021).”
- Large Language Models Can Self-Improve (2022, neutral): “Lightman et al. (2023) treat explanations as process supervision to train a reward model.”
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, methodology): “We can explore process supervision methods like those used in PRM-800K (Lightman et al., 2023), or investigate distillation techniques on successful trajectories, similar to approaches in ToolFormer (Schick et al., 2024) and DSPy (Khattab et al., 2023).”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, background): “For example, even the PRM800K datasets (Lightman et al., 2023), which have been carefully annotated by well-trained annotators, still contain approximately 20% of incorrectly annotations 7 .”
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, neutral): “For instance, DeepSeek-R1 [33] undergoes multiple rounds of SFT and RL training, emphasizing structured thinking templates and rule-based reward mechanisms.”
- ReFT: Reasoning with Reinforced Fine-Tuning (2024, methodology): “Additionally, Lightman et al. (2023) suggests that a well-trained process-based reward model (PRM) can significantly enhance performance.”
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, neutral): “In fact, many “reasoning” methods for LLMs are evaluated only in the math domain; for instance, Lightman et al. (2024) frame their paper as “complex multi-step reasoning” and Mixtral-Large2’s release 2 cited effort “enhancing the model’s reasoning capabilities”, but performance is only re-ported on…”
- OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “MathMix is another OpenAI dataset used to finetune GPT-4 (Lightman et al., 2023) that contains 1B high quality mathematical tokens containing both natural and synthetic data.”
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “…such as Chain-of-Thought (CoT) prompting methods [3], plan-based methods like Tree-of-Thought and Graph-of-Thought [4, 5], process-based reward models [6– 9], Monte Carlo Tree Search (MCTS) and Beam Search [10– 12], as well as constructing complex Supervised Fine-Tuning (SFT) datasets [13, 14].”
- Large Language Model Alignment: A Survey (2023, background): “• Process Supervision (Lightman et al., 2023): Unlike factored cognition, process supervision fragments a complex task into a series of sequential subtasks, each with its own dependencies.”
- UltraFeedback: Boosting Language Models with High-quality Feedback (2023, positive): “…pools to guarantee the diversity of comparison data, then query GPT-4 with detailed illustrations for fine-grained and high-quality annotations in both textual and numerical formats. scenarios (Ji et al., 2023), and math problems (Lightman et al., 2023)), thus cannot boost general chat models.”
- ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (2023, methodology): “Since the SingleEQ, SingleOP, AddSub, and MultiArith datasets focus on different aspects of basic arithmetic, we report their average results under the collective term MAWPS (Koncel-Kedziorski et al., 2016) for all methods.”
- Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs (2025, positive): “These approaches range from simple parallel sampling (Li et al., 2024; Brown et al., 2024) to more sophisticated methods using verifiers or process reward models (PRMs) (Lightman et al., 2023; Yao et al., 2023).”
- VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, neutral): “Furthermore, SSR embodies the principles of curriculum learning [Team et al., 2025] in an online and active fashion Lightman et al. [2023], by dynamically adjusting the training focus towards high-value experiences situated near the model’s decision boundaries.”
- Large Language Models Cannot Self-Correct Reasoning Yet (2023, background): “Cobbe et al. (2021); Lightman et al. (2023); Wang et al. (2023) train a verifier or a critique model on a high-quality dataset to verify or refine LLMs’ outputs.”
- Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, background): “A n s w e r C ARP (Zhang et al., 2023a) 4.9K M Chinese GSM8K (Cobbe et al., 2021) 8.5K M Linguistically diverse MATH (Hendrycks et al., 2021) 12.5K H Problems are put into difficulty levels 1-5 PRM800K (Lightman et al., 2023) 12K H MATH w/ step-wise labels M ATH QA (Amini et al., 2019) 37K C GRE exa”
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “For example, humans may ask models to critique the outputs of other models (Irving et al., 2018; Saunders et al., 2022) or use models to help decompose a problem into simpler sub-problems (Leike et al., 2018; Christiano et al., 2018; Lightman et al., 2023).”
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “…Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVista [153]; (3) Text-Centric Tasks : MATH500 [65], AIME24 [84], AIME25 [85], GPQA [106], [31].”