Software Testing With Large Language Models: Survey, Landscape, and Vision

By Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, Qing Wang. IEEE Transactions on Software Engineering, 2023

664 citations recorded by the source corpus.

Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/f43b8a87a96f8abc2467b90538b643a6061416e9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [147].”
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2023, neutral): “Section 2 introduces the main procedure of our study.”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, background): “(2018), testing (Kang et al., 2023; Xia et al., 2023; Wang et al., 2023), and program repair (Monperrus, 2018; Gupta et al.”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Further evidence that there may be a growing issue with scientific evaluation quality in the literature on LLM-Based Software Engineering can be found in the survey of LLM-Based Testing by Wang et al. [94].”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, methodology): “This phase primarily involved excluding papers that mentioned LLMs but did not directly apply them, such as papers that only discussed LLMs in future work or focused on evaluating the performance of LLM-enabled tools [328] (Exclusion criteria 8).”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, neutral): “Therefore, we still consider LLM-based test generation [56] promising, but with proper designs.”
Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, methodology): “This includes software testing, where Wang et al. have presented a survey on the use of LLMs in the field [114].”