WizardCoder: Empowering Code Large Language Models with Evol-Instruct
By Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang. 2023
1,012 citations recorded by the source corpus.
Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated exceptional performance in code-related tasks. However, most existing models are solely pre-trained on extensive raw code data without instruction fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Through comprehensive experiments on four prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we unveil the exceptional capabilities of our model. It surpasses all other open-source Code LLMs by a substantial margin. Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+. Our code, model weights, and data are public at https://github.com/nlpxucan/WizardLM
Subfields: Artificial Intelligence, Information Systems, Alignment & RLHF, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
- Hongsheng Li
In Information Systems
In Alignment & RLHF
In Code Generation & Program Synthesis
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…domains and tasks such as program repair (Zheng et al., 2024; Olausson et al., 2023), optimization (Madaan et al., 2023a), test generation (Steenhoek et al., 2023), documentation generation (Luo et al., 2024), tool usage (Patil et al., 2023; Qin et al., 2024), SQL (Sun et al., 2023), and more.”
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “5B (row 5) with that of its instruction-tuned version WizardCoder-15.”
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, methodology): “Our goal is to comprehensively evaluate recent and widely used LLMs, both specialized for code generation [46, 70, 18, 2, 52, 38, 64] and general-purpose tasks [49, 48, 12, 60, 63, 5, 26].”
- Aligning Large Language Models with Human: A Survey (2023, methodology): “WizardCoder (Luo et al., 2023) 15B EN, Code StarCoder SFT GPT-3.”
- Competition-Level Problems are Effective LLM Evaluators (2023, background): “Meanwhile, research on open-source code LLMs is also thriving, such as CodeLlama (Rozière et al., 2023), StarCoder (Li et al., 2023), CodeGeeX (Zheng et al., 2023), CodeFuse (Di et al., 2023), WizardCoder (Luo et al., 2023) and Lemur (Xu et al., 2023).”
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (2023, background): “The WizardCoder series [16] performs least effectively in both original and counterfactual contexts.”
- How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, background): “1”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, neutral): “Significant research focuses on Mathematical Word Problems (MWP), which demand understanding of mathematical concepts and multi-step reasoning (Zheng et al., 2023; Zhao et al., 2023; Yuan et al., 2023a).”
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, background): “Specifically, model tuning may still be necessary for tasks such as coding (Luo et al., 2023), mathematics (Yue et al., 2023), interactive agents (Yin et al., 2023), etc.”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “Instruction-tuned code models [40, 38, 59], are generally better at programming than their base ones.”
- LoRA Learns Less and Forgets Less (2024, methodology): “It reproduces the “Evol-Instruct” dataset of WizardCoder (Luo et al., 2023b) by iteratively prompting an LLM (GPT-4) to increase the difficulty of a set of question-answer pairs from Code Alpaca (Chaudhary, 2023).”
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “…finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li et al. (2023b); Zelikman et al. (2022).”
- VerilogEval: Evaluating Large Language Models for Verilog Code Generation (2023, background): “WizardLM’s Evol-Instruct [18] evolves instructions for a diverse dataset, which is further applied to code generation [19].”
- AgentBench: Evaluating LLMs as Agents (2023, neutral): “0 Microsoft wizardcoder-15b [48] 15B open v1.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “Complex Instructions Evol-Instruct [144], [145] prompts”
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, methodology): “Third, we have observed significant advancements in code generation models, such as GPT-4 (OpenAI, 2023), StarCoder (Li et al., 2023), and WizardCoder (Luo et al., 2023).”