Large Language Models for Software Engineering: A Systematic Literature Review

By Xinying Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John C. Grundy. ACM Transactions on Software Engineering and Methodology, 2023

1,174 citations recorded by the source corpus.

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/000f964393dafe113a8e66734d63b2a145844159

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “…finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li et al. (2023b); Zelikman et al. (2022).”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “For example, Hou et al. [14] provided an excellent recent SLR covering 229 research papers from 2017 to 2023 reporting SE tasks tackled, data collection and preprocessing techniques, and strategies for optimising LLM performance (such as prompt engineering).”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “Automated software engineering, like code generation and repair [19, 24, 72], has attracted significant interest from academia and industry to facilitate the development of advanced models.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “Hou et al. conducted a systematic literature review on using LLMs for software engineering, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes [164].”