Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models

By Sami Sarsa, Paul Denny, Arto Hellas, Juho Leinonen. 2022

502 citations recorded by the source corpus.

This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.

Subfields: Computer Science Applications, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

  1. Paul Denny
  2. Juho Leinonen
  3. Arto Hellas
  4. Sami Sarsa
  5. Stephen MacNeil
  6. Seth Bernstein
  7. Andrew Tran
  8. Joanne Kim

In Information Systems

  1. Paul Denny
  2. Juho Leinonen
  3. Arto Hellas
  4. Sami Sarsa
  5. Shin Yoo
  6. Mark Harman
  7. Stephen MacNeil
  8. Seth Bernstein

In Software

  1. Shin Yoo
  2. Mark Harman
  3. Paul Denny
  4. Juho Leinonen
  5. Arto Hellas
  6. Sami Sarsa
  7. Shubho Sengupta
  8. Mitya Lyubarskiy

In Large Language Models

  1. Paul Denny
  2. Juho Leinonen
  3. Nasser Giacaman
  4. Viraj Kumar
  5. Sami Sarsa
  6. Arto Hellas
  7. Jian-Guang Lou
  8. Zeqi Lin

In Code Generation & Program Synthesis

  1. Paul Denny
  2. Juho Leinonen
  3. Sami Sarsa
  4. Arto Hellas
  5. Stephen MacNeil
  6. Seth Bernstein
  7. Andrew Tran
  8. Joanne Kim

Citations in the corpus