Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book

By Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, Juho Leinonen. 2022

251 citations recorded by the source corpus.

Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.

Subfields: Computer Science Applications, Information Systems, Software, Code Generation & Program Synthesis, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Software

In Code Generation & Program Synthesis

In Human-Computer Interaction

Citations in the corpus

Comparing Code Explanations Created by Students and Large Language Models (2023, methodology): “MacNeil et al. used the GPT-3 model to generate explanations of short code fragments which then were presented to students in an online e-book alongside the corresponding code [24].”
Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “Initial evaluation by MacNeil et al. [80] on their interactive Web development e-book, suggested that a majority of students perceived LLM-generated code explanations to be helpful.”
Explaining Competitive-Level Programming Solutions using LLMs (2023, background): “Several existing works have explored generating code explanations using LLMs. MacNeil et al. (2023) integrated LLM-generated code explanations into an interactive e-book on web software development, showing that students found the generated explanations helpful.”
Computing Education in the Era of Generative AI (2023, methodology): “This follows a similar evaluation approach employed by the Codex developers [5].”
The promise and challenges of generative AI in education (2024, unknown): “In particular, following early promising results, several teams attempted to leverage LLMs’ code explanation ability to produce working code examples, that is, examples of programming problem solutions augmented with code explanations (Hassany et al. 2023; Jury et al. 2024; MacNeil et al. 2023).”