Self-Collaboration Code Generation via ChatGPT

By Yihong Dong, Xue Jiang, Zhi Jin, Ge Li. ACM Transactions on Software Engineering and Methodology, 2023

528 citations recorded by the source corpus.

Although large language models (LLMs) have demonstrated remarkable code-generation ability, they still struggle with complex tasks. In real-world software development, humans usually tackle complex tasks through collaborative teamwork, a strategy that significantly controls development complexity and enhances software quality. Inspired by this, we present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT. Specifically, through role instructions, (1) Multiple LLM agents act as distinct “experts,” each responsible for a specific subtask within a complex task; (2) Specify the way to collaborate and interact, so that different roles form a virtual team to facilitate each other’s work, ultimately the virtual team addresses code generation tasks collaboratively without the need for human intervention. To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework. Thus, we assemble an elementary team consisting of three LLM roles (i.e., analyst, coder, and tester) responsible for software development’s analysis, coding, and testing stages. We conduct comprehensive experiments on various code-generation benchmarks. Experimental results indicate that self-collaboration code generation relatively improves 29.9–47.1% Pass@1 compared to the base LLM agent. Moreover, we showcase that self-collaboration could potentially enable LLMs to efficiently handle complex repository-level tasks that are not readily solved by the single LLM agent.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/ba2f935d2578fbf77ec1aa79e26e3db396771e38

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, background): “[ Dong et al. , 2023b] makes LLMs work as distinct “experts” for sub-tasks in software development, autonomously collaborating to generate code.”
Cognitive Architectures for Language Agents (2023, unknown): “Digital environments .”
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023, unknown): “(A1, A2, A5, and A6) • Collaborative/adversarial agent interactions : Like many collaborative agent systems (Dong et al., 2023), agents in AutoGen can share information and knowledge, to complement each other’s abilities and collectively arrive at better solutions.”
Communicative Agents for Software Development (2023, methodology): “(3) Software implementation is the detailed creation procedure of the software to implement the design [16; 1; 6; 29; 11].”
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework (2023, background): “, [14]), agents in AutoGen can share information and knowledge, allowing them to complement each other’s abilities and collectively arrive at better solutions.”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Dong et al. [72] treated LLMs as agents, letting multiple LLMs play distinct roles in addressing code generation tasks collaboratively and interactively.”
A survey on large language model based autonomous agents (2023, unknown): “ChemCrow [75], Boiko et al. [115], Grossmann et al. [122] Natural Science Education ChemCrow [75], CodeHelp [120], Boiko et al. [115], MathA-gent [117], Drori et al. [118] Engineering”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “LLMs have been used for various coding-related tasks including code generation and code recommendation [20], [21], [22], [23].”
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, background): “…recent efforts have been made to improve the effectiveness of code generation models by incorporating in-context learning and its variations [ Dong et al. , 2023b; Wei et al. , 2022; Le et al. , 2023; Huang et al. , 2023; Zhang et al. , 2023b; Chen et al. , 2023b; Madaan et al. , 2023 ] ,…”
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “”) to inspire ChatGPT’s capability of test generation, which is a common prompt optimization strategy [53], [64]; and (ii) a task-description instruction (i.”