Repository-Level Prompt Generation for Large Language Models of Code

By Disha Shrivastava, Hugo Larochelle, Daniel Tarlow. 2022

193 citations recorded by the source corpus.

With the success of large language models (LLMs) of code and their use as code assistants (e.g. Codex used in GitHub Copilot), techniques for introducing domain-specific knowledge in the prompt design process become important. In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using prompt proposals. The prompt proposals take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g. imports, parent class files). Our technique doesn't require any access to the weights of the LLM, making it applicable in cases where we only have black-box access to the LLM. We conduct experiments on the task of single-line code-autocompletion using code repositories taken from Google Code archives. We demonstrate that an oracle constructed from our prompt proposals gives a remarkably high relative improvement of 36% over Codex, showing the quality of these proposals. Further, we show that when we train a model to predict a prompt proposal, we can achieve significant performance gains over Codex and other baselines. We release our code, data, and trained checkpoints at: \url{https://github.com/shrivastavadisha/repo_level_prompt_generation}.

Subfields: Artificial Intelligence, Information Systems, Information Systems and Management, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Systems and Management

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “…engineering tasks, such as JuICe (Agashe et al., 2019), APIBench (Patil et al., 2023), RepoBench (Liu et al., 2023c), ODEX (Wang et al., 2022b), SWE-Bench (Jimenez et al., 2023), GoogleCodeRepo (Shrivastava et al., 2023), RepoEval (Zhang et al., 2023a), and Cocomic-Data (Ding et al., 2022).”
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair (2024, unknown): “…for code generation and code editing: Beyond program repair, LLMs have been applied to a variety of other code generation and code editing tasks, including code completion [35], [59], fuzzing [60], generating and improving unit tests [61], [62], [63], [64], [65], [66], multi-step code editing [67].”
Learning Performance-Improving Code Edits (2023, background): “Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (2023, methodology): “The scope of code completion has expanded with works like RLPG [36], CoCoMIC [12], and RepoCoder [48], emphasizing the integration of in-file and cross-file contexts and the importance of specialized benchmarks for evaluating repository-level code autocompletion systems.”
RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, methodology): “To determine the original source of these code snippets, we adopt a classification scheme inspired by Shrivastava et al. (2022), consisting of five distinct file locations: 1.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “In other cases, [124] introduced the Repo-Level Prompt Generator (RLPG) to dynamically retrieve relevant repository context and construct a prompt for a given task.”
A Survey on Large Language Models for Code Generation (2024, unknown): “Unfortunately, these measures are not suitable for evaluating code generation because functional correctness is much more important.”