The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
By James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, James Prather. 2022
509 citations recorded by the source corpus.
Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities.
Subfields: Computational Theory and Mathematics, Computer Science Applications, Information Systems, Code Generation & Program Synthesis, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Theory and Mathematics
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- Brendan Dolan-Gavitt
- Siddharth Garg
- Foutse Khomh
- Ramesh Karri
In Computer Science Applications
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Information Systems
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Code Generation & Program Synthesis
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Human-Computer Interaction
- Paul Denny
- Sami Sarsa
- Juho Leinonen
- Arto Hellas
- Seth Bernstein
- Andrew Tran
- Stephen MacNeil
- Joanne Kim
Citations in the corpus
- Comparing Code Explanations Created by Students and Large Language Models (2023, methodology): “A seminal study in this field, entitled “The Robots are Coming” [13], utilized the Codex model and a private repository of programming problems drawn from high-stakes summative assessments.”
- Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “These LLMs can write functionally correct code, with studies showing capabilities in solving introductory programming tasks [17, 18] and algorithmic challenges [19]).”
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming (2023, unknown): “[30] showed that OpenAI Codex outperforms most students on code writing questions that are used in typical first-year programming exams.”
- Robosourcing Educational Resources - Leveraging Large Language Models for Learnersourcing (2022, background): “In the context of solving programming exercises, Finnie-Ansley et al. [42] observed that Codex was able to correctly answer most introductory programming problems and that when given typical introductory programming exam questions, Codex performed better than the average student.”
- A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “problems, explaining code snippets, and generating course materials (Finnie-Ansley et al., 2022; MacNeil et al., 2022).”
- xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “…et al., 2022), program repair (Berabi et al., 2021), code translation (Roziere et al., 2020, 2021b), and code retrieval (Wan et al., 2019; Parvez et al., 2021) are having a profound impact on increasing developer productivity (Ziegler et al., 2022) and aiding educators (Finnie-Ansley et al., 2022).”
- Do Users Write More Insecure Code with AI Assistants? (2022, background): “Ensuring that cryptography library defaults are secure, educating users on how to interact with and test an AI assistant ( [7]), and providing integrated warnings and potential validation tests based on the generated code ( [2]) are important solutions to consider as AI code assistants become more c”
- Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book (2022, methodology): “Previous research has demonstrated how LLMs such as Codex can generate code for students based on a specification [11] as well as generate assignments for instructors [36].”
- Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation (2022, background): “This was illustrated by Finnie-Ansley et al. who observed a great deal of variation in the solutions that Codex generated when solving the classic Rainfall problem [12].”
- Computing Education in the Era of Generative AI (2023, background): “This can facilitate discussions about different approaches and the quality of solutions, and provide opportunities for refactoring exercises [15].”
- Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, methodology): “Our work suggests that modern machine learning models such as OpenAI Codex provide many opportunities for programming course designers, although potential challenges outlined in prior work [27] should not be ignored.”
- The promise and challenges of generative AI in education (2024, unknown): “Computer science education (CSEd) could be considered as a special domain for the educational application of LLMs.”
- GitHub Copilot AI pair programmer: Asset or Liability? (2022, methodology): “Finnie-Ansley et al. [21] used Davinci (an API on a beta version of Codex) on two datasets.”
- Repairing Bugs in Python Assignments Using Large Language Models (2022, unknown): “AI, both symbolic and neural, has been extensively applied to the domain of education [39], [40].”
- Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (2022, background): “Nevertheless, across the 166 problems we evaluated, overall we observe similar rates of initial success reported in earlier studies [11].”