An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
By Max Schäfer, Sarah Nadi, Aryaz Eghbali, Frank Tip. 2023
557 citations recorded by the source corpus.
Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. Large Language Models (LLMs) have recently been applied to various aspects of software development, including their suggested use for automated generation of unit tests, but while requiring additional training or few-shot learning on examples of existing tests. This paper presents a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation without requiring additional training or manual effort. Concretely, we consider an approach where the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. Furthermore, if a generated test fails, our approach attempts to generate a new test that fixes the problem by re-prompting the model with the failing test and error message. We implement our approach in <sc>TestPilot</sc>, an adaptive LLM-based test generation tool for JavaScript that automatically generates unit tests for the methods in a given project's API. We evaluate <sc>TestPilot</sc> using OpenAI's <italic>gpt3.5-turbo</italic> LLM on 25 npm packages with a total of 1,684 API functions. The generated tests achieve a median statement coverage of 70.2% and branch coverage of 52.8%. In contrast, the state-of-the feedback-directed JavaScript test generation technique, Nessie, achieves only 51.3% statement coverage and 25.6% branch coverage. Furthermore, experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. We also find that 92.8% of <sc>TestPilot</sc>'s generated tests have <inline-formula><tex-math notation="LaTeX">$\leq$</tex-math><alternatives><mml:math display="inline"><mml:mo>≤</mml:mo></mml:math><inline-graphic xlink:href="schaefer-ieq1-3334955.gif"/></alternatives></inline-formula> 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies. Finally, we run <sc>TestPilot</sc> with two additional LLMs, OpenAI's older <italic>code-cushman-002</italic> LLM and <italic>StarCoder</italic>, an LLM for which the training process is publicly documented. Overall, we observed similar results with the former (68.2% median statement coverage), and somewhat worse results with the latter (54.0% median statement coverage), suggesting that the effectiveness of the approach is influenced by the size and training set of the LLM, but does not fundamentally depend on the specific model.
Subfields: Artificial Intelligence, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Armando Solar-Lezama
- Koushik Sen
- Naman Jain
- Tianjun Zhang
- Mingwei Liu
- Wen-Ding Li
- Yixuan Chen
- Zhiqiang Yuan
In Information Systems
- Armando Solar-Lezama
- Koushik Sen
- Naman Jain
- Tianjun Zhang
- Ion Stoica
- Rishabh Singh
- Mosharaf Chowdhury
- Charles Sutton
In Software
- Armando Solar-Lezama
- Koushik Sen
- Naman Jain
- Tianjun Zhang
- Ion Stoica
- Rishabh Singh
- Patrice Godefroid
- Gordon Fraser
In Large Language Models
In Code Generation & Program Synthesis
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Test generation using LLMs has been explored in (Yuan et al., 2023; Sch¨afer et al., 2024; Tufano et al., 2022; Watson et al., 2020).”
- No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “…effectiveness of C HAT T ESTER and the individual contribution of each component ( i.e., the initial test generator and the iterative test refiner) in C HAT T ESTER , we study three techniques: • ChatGPT: the default ChatGPT with the basic prompt, which is the one used in our empirical study.”
- Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “For example, in their work for unit test case generation, Alagarsamy et al. [29] additionally introduce a verification method to check and repair the naming consistency (i.e., revising the test method name to be consistent with the focal method name) and the test signatures (i.e., adding missing…”
- Self-Collaboration Code Generation via ChatGPT (2023, unknown): “(2) This setting is mainly suited for function-level code generation and proves challenging to extend to file-level or repository-level code generation.”
- Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “As a result, LLMs have shown promise in automating or assisting with software engineering [127] and software testing [114] tasks, including test generation [97], documentation [42], refactoring [118], and automated program repair [104].”
- CLARC: C/C++ Benchmark for Robust Code Search (2026, unknown): “LLMs help constructing critical evaluation components, including natural language instructions (Zhu et al., 2024), code solutions (Ahmad et al., 2025), and test cases (Schäfer et al., 2024; Alshahwan et al., 2024).”