No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

By Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng. 2023

211 citations recorded by the source corpus.

Unit testing is essential in detecting bugs in functionally-discrete program units. Manually writing high-quality unit tests is time-consuming and laborious. Although traditional techniques can generate tests with reasonable coverage, they exhibit low readability and cannot be directly adopted by developers. Recent work has shown the large potential of large language models (LLMs) in unit test generation, which can generate more human-like and meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning and reinforcement learning, has performed well in various domains. However, It remains unclear how effective ChatGPT is in unit test generation. In this work, we perform the first empirical study to evaluate ChatGPT's capability of unit test generation. Specifically, we conduct a quantitative analysis and a user study to systematically investigate the quality of its generated tests regarding the correctness, sufficiency, readability, and usability. The tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures. Still, the passing tests generated by ChatGPT resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTESTER incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.

Subfields: Computer Networks and Communications, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Test generation using LLMs has been explored in (Yuan et al., 2023; Sch¨afer et al., 2024; Tufano et al., 2022; Watson et al., 2020).”
Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “Nie et al. [95] report 29% of tests generated using TeCo are executable, while Yuan et al. [96] found that approximately one-quarter of the tests generated by ChatGPT were executable, rising to one-third with suitable prompt engineering.”
Fuzz4ALL: Universal Fuzzing with Large Language Models (2023, methodology): “In addition to fuzzing, LLMs have also been applied to the re-lated problem of unit test generation [5, 39, 54, 66, 74, 82].”
Evaluating Large Language Models in Class-Level Code Generation (2024, methodology): “These include code generation [46, 54, 62, 72], code summarization [12, 51] and various software maintenance tasks, including vulnerability detection [58, 61, 73], test generation [68, 69], and program repair [26, 36, 37, 63].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Several papers [33, 6, 229, 305, 149, 244, 324] discuss the utilization of LLMs for generating test cases, with our particular emphasis on those addressing security implications.”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation (2023, unknown): “Previous research on unit test generation has employed evolutionary algorithms, leading to tools such as EvoSuite [15] and Randoop [27], and several notable machine learning models [47, 3, 19, 13, 40, 39, 49, 51, 12, 28, 14, 34, 18].”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “On one hand, in software testing, there are many tasks related to code generation, such as unit test generation [7], where the utilization of LLMs is expected to yield good performance.”
Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, background): “However, an agent setup where the LLM is automatically provided with compilation errors or where test creation is divided into sub-tasks such as planning and understanding can overcome this challenge [125].”