The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming

By James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, James Prather. 2022

509 citations recorded by the source corpus.

Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities.

Subfields: Computational Theory and Mathematics, Computer Science Applications, Information Systems, Code Generation & Program Synthesis, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

  1. Paul Denny
  2. Juho Leinonen
  3. Sami Sarsa
  4. Arto Hellas
  5. Brendan Dolan-Gavitt
  6. Siddharth Garg
  7. Foutse Khomh
  8. Ramesh Karri

In Computer Science Applications

  1. Paul Denny
  2. Juho Leinonen
  3. Arto Hellas
  4. Sami Sarsa
  5. Stephen MacNeil
  6. Seth Bernstein
  7. Andrew Tran
  8. Joanne Kim

In Information Systems

  1. Paul Denny
  2. Juho Leinonen
  3. Arto Hellas
  4. Sami Sarsa
  5. Stephen MacNeil
  6. Seth Bernstein
  7. Andrew Tran
  8. Joanne Kim

In Code Generation & Program Synthesis

  1. Paul Denny
  2. Juho Leinonen
  3. Sami Sarsa
  4. Arto Hellas
  5. Stephen MacNeil
  6. Seth Bernstein
  7. Andrew Tran
  8. Joanne Kim

In Human-Computer Interaction

  1. Paul Denny
  2. Sami Sarsa
  3. Juho Leinonen
  4. Arto Hellas
  5. Seth Bernstein
  6. Andrew Tran
  7. Stephen MacNeil
  8. Joanne Kim

Citations in the corpus