Repairing Bugs in Python Assignments Using Large Language Models

By Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, R. Piskac, Gustavo Soares, Gust Verbruggen. 2022

67 citations recorded by the source corpus.

Students often make mistakes on their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used to synthesize such fixes. Prior work has explored the use of symbolic and neural techniques for APR in the education domain. Both types of approaches require either substantial engineering efforts or large amounts of data and training. We propose to use a large language model trained on code, such as Codex, to build an APR system -- MMAPR -- for introductory Python programming assignments. Our system can fix both syntactic and semantic mistakes by combining multi-modal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. We evaluate MMAPR on 286 real student programs and compare to a baseline built by combining a state-of-the-art Python syntax repair engine, BIFI, and state-of-the-art Python semantic repair engine for student assignments, Refactory. We find that MMAPR can fix more programs and produce smaller patches on average.

Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, methodology): “…tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022;…”
Large Language Models for Education: A survey and outlook (2024, unknown): “For example, Zhang et al. [5] leverage Codex, a code-trained LLM, to build an automatic program repair system for introductory Python assignments.”
RLTF: Reinforcement Learning from Unit Test Feedback (2023, background): “This can result in RL training quality being limited by the diversity of the dataset and hinder the effective exploration of the environment (Fujimoto et al., 2019; Kumar et al., 2019).”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “Existing studies [14], [105], [106], [114] have shown that LLM can refactor and improve existing codes.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “…and involve different inputs, including (i) inputting a buggy function with the goal of outputting the patched function, (ii) inputting the buggy location with the goal of generating the correct replacement code (can be a single line change) given the preﬁx and sufﬁx of the buggy function [93].”