A Survey on Large Language Models for Code Generation

By Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, Sunghun Kim. ACM Transactions on Software Engineering and Methodology, 2024

1,059 citations recorded by the source corpus.

Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot. Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of Natural Language Processing (NLP) or Software Engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and provide a quantitative and qualitative comparative analysis of experimental results of code LLMs, sourced from their original papers to ensure a fair comparison on the HumanEval, MBPP, and BigCodeBench benchmarks, across various levels of difficulty and types of programming tasks, to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page (https://github.com/juyongjiang/CodeLLMSurvey) to continuously document and disseminate the most recent advances in the field.

Subfields: Artificial Intelligence, Health Informatics, Information Systems

View Paper: https://www.semanticscholar.org/paper/c8b18682965ff9dccc0130dab3d679f78cefa617

Citations in the corpus

• A Survey on Mixture of Experts in Large Language Models (2024, neutral): “…I N THE current landscape of artificial general intelligence (AGI), the transformative impact of transformer-based large language models (LLMs) has permeated diverse fields such as natural language processing [1], [2], [3], [4], [5], computer vision [6], [7], and multimodality [8], [9], [10].”
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “Fueled by increasingly good code completion models (Jiang et al., 2024a), downstream applications have quickly grown in popularity following the emergence of code assistants.”
• RUMAD: Reinforcement-Unifying Multi-Agent Debate (2026, unknown): “Multi-agent debate (MAD) frameworks harness the collective intelligence of multiple large language models (LLMs) [1, 5, 6, 9, 13, 21, 25] to achieve superior reasoning and decision-making beyond what any single model can attain.”
• CRANE: Reasoning with constrained LLM generation (2025, unknown): “…systems that interact with traditional software tools like Python interpreters (OpenAI, 2024; Chen et al., 2023) for code generation (Suresh et al., 2024a;b; Jiang et al., 2024), logical solvers (Pan et al., 2023; Olaus-* son et al., 2023), and theorem provers (Wu et al., 2022; Yang et al., 2023).”
• CLARC: C/C++ Benchmark for Robust Code Search (2026, unknown): “Although recent Large Language Models (LLMs) and Code Language Models (CLMs) demonstrate strong reasoning capabilities for generation tasks (Jiang et al., 2024; Rozière et al., 2024; Hui et al., 2024; Chen et al., 2021), deploying such heavy architectures for large-scale retrieval is often…”
• Agentic Software Issue Resolution with Large Language Models: A Survey (2025, unknown): “Following prior surveys [63, 73, 98, 102], we adopt the systematic approach proposed by Kitchenham et al [85].”