Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
By Sami Sarsa, Paul Denny, Arto Hellas, Juho Leinonen. 2022
502 citations recorded by the source corpus.
This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.
Subfields: Computer Science Applications, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Information Systems
In Software
- Shin Yoo
- Mark Harman
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Shubho Sengupta
- Mitya Lyubarskiy
In Large Language Models
In Code Generation & Program Synthesis
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
Citations in the corpus
- Comparing Code Explanations Created by Students and Large Language Models (2023, methodology): “This is also the type of explanation that LLMs seem to be best at creating [36].”
- Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “These LLMs can write functionally correct code, with studies showing capabilities in solving introductory programming tasks [17, 18] and algorithmic challenges [19]).”
- CodeT: Code Generation with Generated Tests (2022, background): “Codex is proficient in understanding the provided context and generating functional programs, and it has been successfully applied to many programming tasks (Drori et al., 2021; Pearce et al., 2021; Sarsa et al., 2022).”
- Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book (2022, methodology): “Explanations of code can also be generated by LLMs [24, 37].”
- Language Models Can Teach Themselves to Program Better (2022, background): “While LMs have in fact been shown to be capable of generating largely-correct English programming problems [Sarsa et al., 2022], human oversight is still required for vetting the descriptions and test cases.”
- Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (2022, methodology): “applied Codex to the task of generating novel programming exercises given a single example as input [20].”
- CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs (2024, methodology): “In terms of generating learning resources, early work by Sarsa et al. [53] analyzed the novelty, plausibility, and readiness of 120 programming exercises generated by OpenAI Codex and proposed the potential of using such models to come up with coding assignments.”
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming (2023, unknown): “Additionally, these tools could support educators in curriculum development by generating programming exercises and explanations to solutions [79].”
- Using Large Language Models to Enhance Programming Error Messages (2022, background): “Tools like Codex are able to generate new programming assignments [36] and code explanations [29] when provided examples.”
- Practical and ethical challenges of large language models in education: A systematic scoping review (2023, background): “Additionally, for introductory programming courses, the state-of-the-art LLMs, Codex, could generate sensible and novel exercises for students along with an appropriate sample solution (around three out of four times) and accurate code explanation (67% accuracy) (Sarsa et al., 2022).”
- Robosourcing Educational Resources - Leveraging Large Language Models for Learnersourcing (2022, background): “For more details on the evaluation, please see [21].”
- Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation (2022, background): “first-year (CS1 and even CS2) programming assignments and even exams seems to have become much easier [12, 13, 35].”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Several other authors [80], [81], [214] explored the capabilities of CodeX for generating programming exercises and code explanations.”