Using Large Language Models to Enhance Programming Error Messages
By Juho Leinonen, Arto Hellas, Sami Sarsa, Brent N. Reeves, Paul Denny, James Prather, Brett A. Becker. 2022
245 citations recorded by the source corpus.
A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for an audience that already knows how to program, or even for programming environments that then use the information to highlight areas in code. Researchers have been working on making these errors more novice friendly since the 1960s, however progress has been slow. The present work contributes to this stream of research by using large language models to enhance programming error messages with explanations of the errors and suggestions on how to fix them. Large language models can be used to create useful and novice-friendly enhancements to programming error messages that sometimes surpass the original programming error messages in interpretability and actionability. These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students. We further discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.
Subfields: Computer Science Applications, Information Systems, Large Language Models, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Information Systems
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Stephen MacNeil
- Seth Bernstein
- Andrew Tran
- Joanne Kim
In Large Language Models
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- James Prather
- Brett A. Becker
- Brent N. Reeves
- Nasser Giacaman
In Software Engineering
Citations in the corpus
- Comparing Code Explanations Created by Students and Large Language Models (2023, background): “Other work has found that a lower temperature value might work better in the context of code [22].”
- Computing Education in the Era of Generative AI (2023, methodology): “Leinonen et al. recently took a different approach and used LLMs to try to improve PEMs [21].”
- The promise and challenges of generative AI in education (2024, unknown): “For tasks such as code explanation generation, success and correctness rates might be higher, and newer models tend to perform better than older models (i.e. Codex vs. GPT 3.5 vs. GPT 4) on most types of tasks (Savelka et al. 2023).”