Comparing Code Explanations Created by Students and Large Language Models

By Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, Arto Hellas. 2023

209 citations recorded by the source corpus.

Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

Subfields: Artificial Intelligence, Information Systems, Information Systems and Management, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Systems and Management

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

StarCoder: may the source be with you! (2023, background): “Code LLMs We expect Code LLMs to enable people from diverse backgrounds to learn to write higher-quality code and develop low-code applications (Leinonen et al., 2023).”
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “…(Svajlenko et al., 2014; Lu et al., 2021), Defect-detection (Zhou et al., 2019; Li et al., 2021; Lu et al., 2021), Code-explaination (Muennighoff et al., 2023; Leinonen et al., 2023), and Question Answering over Code (Huang et al., 2021; Liu & Wan, 2021; Lee et al., 2022; Sahu et al., 2024).”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023, background): “The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems [34, 82, 115, 118, 158] and assist in debugging code [116, 140, 187].”
Explaining Competitive-Level Programming Solutions using LLMs (2023, background): “Leinonen et al. (2023) compared LLM-generated explana- tions with student-created explanations, finding that the LLM-created explanations were easier to understand and more accurate.”
Computing Education in the Era of Generative AI (2023, background): “Recent work has also found that LLM-created code explanations are rated more highly on average by students than code explanations created by their peers [20].”