Evaluating Large Language Models Trained on Code

By Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yuri Burda, Yura Burda and 52 more. arXiv.org, 2021

10,663 citations recorded by the source corpus.

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

Subfields: Artificial Intelligence, Hardware and Architecture, Information Systems, Evaluation & Benchmarks

View Paper: https://arxiv.org/abs/2107.03374

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Percy Liang
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Quoc V. Le
  6. Yoshua Bengio
  7. Graham Neubig
  8. Yejin Choi

In Hardware and Architecture

  1. Quoc V. Le
  2. Tri Dao
  3. Josh Tenenbaum
  4. Tim Dettmers
  5. Pushmeet Kohli
  6. Dawn Song
  7. Armando Solar-Lezama
  8. Gabriel Recchia

In Information Systems

  1. Percy Liang
  2. Uri Alon
  3. Miltiadis Allamanis
  4. Charles Sutton
  5. Jamie Callan
  6. Jonathan Berant
  7. Hannaneh Hajishirzi
  8. Graham Neubig

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Yejin Choi
  3. Hannaneh Hajishirzi
  4. Luke S. Zettlemoyer
  5. Ashish Sabharwal
  6. Matt Gardner
  7. Douwe Kiela
  8. Jianfeng Gao

Citations in the corpus