Training Verifiers to Solve Math Word Problems
By Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano and 2 more. 2021
9,717 citations recorded by the source corpus.
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
Subfields: Artificial Intelligence, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Jie Huang
- Hannaneh Hajishirzi
- Wenhu Chen
- Kevin Chen-Chuan Chang
- Peter Clark
- Tushar Khot
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “An example of chain-of-thought prompting for the GSM8K benchmark (grade-school math problems) is given below in Figure 8.”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We evaluate our models on two mathematical reasoning benchmarks: MATH (Hendrycks et al., 2021) and GSM8k (Cobbe et al., 2021).”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average of the GSM8K (8 shot) (Cobbe et al., 2021) and MATH (4 shot) (Hendrycks et al.”
- Mistral 7B (2023, neutral): “…[24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU…”
- Mixtral of Experts (2024, background): “…[30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16]…”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, methodology): “● Models have the ability to generate neutral language intermediate steps by training from scratch (Ling et al., 2017) or finetuning a pretrained model (Cobbe et al., 2021).”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…(Komeili et al., 2022; Thoppilan et al., 2022; Lazaridou et al., 2022; Shuster et al., 2022; Yao et al., 2022), web browsers (Nakano et al., 2021), calculators (Cobbe et al., 2021; Thoppilan et al., 2022), translation systems (Thoppilan et al., 2022) and Python interpreters (Gao et al., 2022).”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Self-consistency is far simpler than prior approaches that either train an additional verifier (Cobbe et al., 2021) or train a re-ranker given additional human annotations to improve generation quality (Thoppilan et al.”
- Competition-level code generation with AlphaCode (2022, background): “Similar to our sampling and filtering, though on a much smaller scale, Chen et al. (2021), Austin et al. (2021), and Cobbe et al. (2021) found that repeated sampling on the same problem significantly increases the probability of finding a correct solution.”
- Emergent Abilities of Large Language Models (2022, positive): “A recent prompting strategy called chain-of-thought prompting enables language models to solve such problems by guiding them to produce a sequence of intermediate steps before giving the final answer (Cobbe et al., 2021; Wei et al., 2022b; Zhou et al., 2022).”
- Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “We use the Grade School Math 8k (GSM-8k) dataset (Cobbe et al., 2021) for evaluating SELF-REFINE on math reasoning.”
- Augmented Language Models: a Survey (2023, methodology): “, 2012), mathematical reasoning (Cobbe et al., 2021), symbolic reasoning (Wei et al.”
- Galactica: A Large Language Model for Science (2022, methodology): “For GSM8k we use the provided training dataset and convert so the calculator steps are performed by writing a Python program, following the <work> format (Cobbe et al., 2021).”
- Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “Known as Zero-Shot-CoT, this prompting strategy improves the zero-shot performance of InstructGPT on MultiArith (Roy & Roth, 2016) from 17.7 to 78.7 and improves performance on GSM8K(Cobbe et al., 2021) from 10.4 to 40.7.”
- Scaling Instruction-Finetuned Language Models (2022, methodology): “These nine datasets include tasks such as arithmetic reasoning (Cobbe et al., 2021), multi-hop reasoning (Geva et al.”
- A General Language Assistant as a Laboratory for Alignment (2021, background): “These results differ from similar recent experiments on math problem solving [CKB21], though they trained on thousands of times less data.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “To measure math-reasoning capabilities of the proposed method, we report results on the GSM8K benchmark Cobbe et al. (2021), which is comprised of a set of middle-school math word problems.”