Evaluating Large Language Models Trained on Code
By Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yuri Burda, Yura Burda and 52 more. arXiv.org, 2021
10,663 citations recorded by the source corpus.
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems, Evaluation & Benchmarks
View Paper: https://arxiv.org/abs/2107.03374
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Quoc V. Le
- Yoshua Bengio
- Graham Neubig
- Yejin Choi
In Hardware and Architecture
- Quoc V. Le
- Tri Dao
- Josh Tenenbaum
- Tim Dettmers
- Pushmeet Kohli
- Dawn Song
- Armando Solar-Lezama
- Gabriel Recchia
In Information Systems
- Percy Liang
- Uri Alon
- Miltiadis Allamanis
- Charles Sutton
- Jamie Callan
- Jonathan Berant
- Hannaneh Hajishirzi
- Graham Neubig
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Ashish Sabharwal
- Matt Gardner
- Douwe Kiela
- Jianfeng Gao
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “In the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets, the model is given an English-language description of a few sentences and a small number of input-output examples, and the goal is to generate a short Python program, usually a single function.”
- Mistral 7B (2023, neutral): “…• Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English…”
- Mixtral of Experts (2024, neutral): “…• Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English…”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2020), as well as the use of large language models (Chen et al., 2021; Austin et al., 2021).”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “Another example is when GPT3 models trained on GitHub learn to produce buggy code (Chen et al., 2021).”
- Solving Quantitative Reasoning Problems with Language Models (2022, methodology): “Furthermore, the Codex model [9] can generate code solutions to MATH problems [11].”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “We use two public engines code-davinci-001 and code-davinci-002 from the Codex series (Chen et al., 2021) to aid reproducibility; • LaMDA-137B (Thoppilan et al., 2022) is a dense left-to-right, decoder-only language model with 137-billion parameters, pre-trained on a mixture of web documents,…”
- Quantifying Memorization Across Neural Language Models (2022, methodology): “For example, the GitHub Copilot model (Chen et al., 2021) reportedly rarely emits memorized code in benign situations, and most memorization occurs only when the model has been prompted with long code excerpts that are very similar to the training data (Ziegler, 2021).”
- Competition-level code generation with AlphaCode (2022, methodology): “, 2020), have successfully generated code that solves simple programming problems in Python (Austin et al., 2021; Chen et al., 2021).”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “For models of O(100B) parameters, recent work has finetuned task-specific models for program synthesis (Austin et al., 2021; Chen et al., 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We use the following baseline models where appropriate (e.g. in settings where they support the language of the evaluation dataset): • mGPT (Shliazhko et al., 2022), GPT-style models trained on 60 languages from Wikipedia and Common Crawl • GPT-Neo (Black et al., 2021), GPT-J-6B (Wang and Komatsuzak”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “Neural models for code generation have either utilized a left-to-right (causal) autoregressive language modeling objective [16, 18] or, as BERT does, a masked language modeling objective [22, 24].”
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “, 2019), a docstring and function signature (Chen et al., 2021), or problem description (Hendrycks et al.”