WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

By Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Dongmei Zhang. 2023

752 citations recorded by the source corpus.

Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. In this paper, we present WizardMath, which enhances the mathematical CoT reasoning abilities of LLMs without using external python tools, by applying our proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) method to the domain of math. Through extensive experiments on two mathematical reasoning benchmarks, namely GSM8k and MATH, we reveal the extraordinary capabilities of our model. Remarkably, WizardMath-Mistral 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency. Furthermore, WizardMath 70B even outperforms GPT-3.5-Turbo, Claude 2, Gemini Pro and GPT-4-early-version. Additionally, our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance. For more details refer to https://github.com/nlpxucan/WizardLM

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, methodology): “All results with a Llama 2 initialization are copied from the literature (Luo et al., 2023; Yu et al., 2023).”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage (Luo et al., 2023; Wang et al., 2023b).”
MAmmoTH2: Scaling Instructions from the Web (2024, neutral): “…by GPT4 SuperNI [Wang et al., 2022] 96K General SFT NLP Datasets Tora [Gou et al., 2023] 16K Math SFT GSM+MATH Synthesis by GPT4 WizardMath [Luo et al., 2023] 96K Math SFT GSM+MATH Synthesis by GPT4 MathInstruct [Yue et al., 2023b] 262K Math SFT Math datasets Synthesis by GPT4 MetaMathQA…”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, methodology): “…set of the original GSM8K or MATH datasets; (iv) Rejection sampling Fine-Tuning (RFT) [79] generates and collects correct reasoning paths as augmented data for fine-tuning; (v) Wizard-Math [43] which generates samples and trains two reward models using ChatGPT to select samples for fine-tuning.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “Recent efforts have been made to investigate non-linear reasoning structures such as trees(Jung et al., 2022; Zhu et al., 2023; Xu, 2023; Xie et al., 2023; Yao et al., 2023; Hao et al., 2023).”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “…for task-specific finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li et al. (2023b);…”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “MUFFIN (Lou et al., 2023) proposes to scale the instruction according to the input by diversifying these tasks with various input facets.”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “Recently, there has been a growing focus on enhancing the reasoning abilities of Large Language Models (LLMs) (Li et al., 2023b; Yuan et al., 2023a; Luo et al., 2023; Mukherjee et al., 2023), because LLMs still lack reasoning skills (Wang et al., 2023b;d; Zheng et al., 2023; Kaddour et al., 2023;…”
JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, neutral): “…can judge LLMs like a human expert, much concurrent work delving into curating training data (Li et al., 2023a; Kim et al., 2023; Cui et al., 2023; Xu et al., 2023; Jiang et al., 2023; Wang et al., 2023) and benchmarks (Lambert et al., 2024; Zheng et al., 2023) with the help of the closed-source…”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Some reward modeling extensions include process reward models (Luo et al., 2023; Lightman et al., 2023) and step-wise reward models (Havrilla et al., 2024b), which are used for reasoning tasks to provide a correctness label for each of the steps in a reasoning chain.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “In the context of supervised fine-tuning (SFT) of LLMs, human-crafted data has proven to be a remarkably effective source that enhances the performance of LLMs on tasks such as code generation (Roziere et al., 2023; Yang et al., 2023) and mathematical reasoning (Yuan et al., 2023; Luo et al., 2023).”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, background): “…(Touvron et al., 2023), Mistral (Jiang et al., 2023), CodeLlama (13B, instruction-tuned)(Roziere et al., 2023), StarCoder (15.5B)(Li et al., 2023), Wizard-Coder (15B, instruction-tuned)(Xu et al., 2023), Magic-Coder (7B)(Wei et al., 2023) (instruction-tuned), DeepSeek-Coder (6.7B)(Bi et al., 2024).”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “These datasets feature a broader spectrum of difficulties, establishing a robust benchmark for the evaluation of general and math-specific LLMs [71, 64, 57, 19, 44].”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “2) fine-tuning : The fine-tuning meth-ods (Yu et al., 2023b; Luo et al., 2023; Yue et al., 2023; Wang et al., 2023b; Gou et al., 2023) can also enhance the mathematical reasoning ability of LLMs.”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, methodology): “…al., 2023a; Feng et al., 2024), self-reflection (Shinn et al., 2023; Madaan et al., 2023; Wang and Zhao, 2023; Yao et al., 2023b), improved math reasoning (Yue et al., 2023; Luo et al., 2023) and fine-tuning via bootstrapping models (Zelikman et al., 2022; Lewkowycz et al., 2022; Li et al., 2023b).”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “For example, works explore this in the context of multihop question answering (Zhao et al., 2023), math (Luo et al., 2023; Uesato et al., 2022), machine translation (Gulcehre et al., 2023).”
s1: Simple test-time scaling (2025, neutral): “The concurrently released r1-32B shows stronger performance than s1-32B while also only using SFT (DeepSeek-AI et al., 2025).”
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, neutral): “Advancements in LLM math reasoning have largely relied on curating high-quality CoT data, with most leading approaches being GPT-distilled, using frontier models like GPT-4 for synthesis [Wang et al., 2024b, Gou et al., 2023, Luo et al., 2023].”