Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction

By Sungmin Kang, Juyeon Yoon, Shin Yoo. International Conference on Software Engineering, 2022

319 citations recorded by the source corpus.

Many automated test generation techniques have been developed to aid developers with writing tests. To facilitate full automation, most existing techniques aim to either increase coverage, or generate exploratory inputs. However, existing test generation techniques largely fall short of achieving more semantic objectives, such as generating tests to reproduce a given bug report. Reproducing bugs is nonetheless important, as our empirical study shows that the number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. Meanwhile, due to the difficulties of transforming the expected program semantics in bug reports into test oracles, existing failure reproduction techniques tend to deal exclusively with program crashes, a small subset of all bug reports. To automate test generation from general bug reports, we propose Libro, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Since LLMs themselves cannot execute the target buggy code, we focus on post-processing steps that help us discern when LLMs are effective, and rank the produced tests according to their validity. Our evaluation of Libro shows that, on the widely studied Defects4J benchmark, Libro can generate failure reproducing test cases for 33% of all studied cases (251 out of 750), while suggesting a bug reproducing test in first place for 149 bugs. To mitigate data contamination (i.e., the possibility of the LLM simply remembering the test code either partially or in whole), we also evaluate Libro against 31 bug reports submitted after the collection of the LLM training data terminated: Libro produces bug reproducing tests for 32% of the studied bug reports. Overall, our results show Libro has the potential to significantly enhance developer efficiency by automatically generating tests from bug reports.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/ce8de3902a583e1eb6062d4453ed90fe6b9d4558

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Constructed datasets [66, 137, 149, 381] are specialized datasets that researchers create by modifying or augmenting collected datasets to better align with their specific research objectives.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “1) Overall Debug Framework: Bui et al. [77] proposes a uniﬁed Detect-Localize-Repair framework based on the LLM for debugging, which ﬁrst determines whether a given code snippet is buggy or not, then identiﬁes the buggy lines, and translates the buggy code to its ﬁxed version.”
Improving Code Generation by Training with Natural Language Feedback (2023, background): “S E ] 28 M a r 2023 from the Internet, which contains a signiﬁcant number of security vulnerabilities (Kang et al., 2022) and bugs (Chen et al., 2021).”
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “For each project, we clone it from GitHub (as of March 25, 2023) and collect its relevant information ( e.g., its creating time and its last commit time).”
Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “For example, the Libro introduced by Kang et al. [108] uses CodeX to generate tests from bug reports that can reproduce failures.”
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair (2024, unknown): “…for code generation and code editing: Beyond program repair, LLMs have been applied to a variety of other code generation and code editing tasks, including code completion [35], [59], fuzzing [60], generating and improving unit tests [61], [62], [63], [64], [65], [66], multi-step code editing [67].”
A survey on large language model based autonomous agents (2023, unknown): “WebShop [168] 1 3 ✓ 07/2022 Social Simulacra [122] 1 2 08/2022 TE [1] 2 08/2022 LIBRO [78] 4 09/2022 ReAct [170] 1 ✓ 10/2022 Out of One, Many [5] 2 2 3 02/2023 DEPS [154] 1 ✓ 02/2023 Jalil et al.”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We refer to this as the repository structure format , which begins with the root folder of the repository and organizes code files or folder names.”
Self-Collaboration Code Generation via ChatGPT (2023, unknown): “In this section, we outline the most relevant directions and associated papers of this work to highlight our research’s innovative and advanced nature.”
Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “A common maintenance task is to add test cases that reproduce bugs, and LLMs have been applied to generate failure-reproducing test cases [55].”
Agentic Software Issue Resolution with Large Language Models: A Survey (2025, unknown): “Given the strong understanding and generation capabilities of LLMs, Libro [79] first proposes using LLMs to automatically generate reproduction tests, applying post-processing to select promising cases.”