Let's Verify Step by Step

By H. Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe. 2023

3,887 citations recorded by the source corpus.

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Graham Neubig
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Quoc V. Le
  7. Hannaneh Hajishirzi
  8. Yejin Choi

In Prompting & Reasoning

  1. Graham Neubig
  2. Sean Welleck
  3. Jie Huang
  4. Michael R Douglas
  5. Xinyun Chen
  6. Pengfei Liu
  7. Zhiyuan Liu
  8. Swaroop Mishra

Citations in the corpus