RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems

By Tianyang Liu, Canwen Xu, Julian McAuley. 2023

404 citations recorded by the source corpus.

Large Language Models (LLMs) have greatly advanced code auto-completion systems, with a potential for substantial productivity enhancements for developers. However, current benchmarks mainly focus on single-file tasks, leaving an assessment gap for more complex, real-world, multi-file programming scenarios. To fill this gap, we introduce RepoBench, a new benchmark specifically designed for evaluating repository-level code auto-completion systems. RepoBench supports both Python and Java and consists of three interconnected evaluation tasks: RepoBench-R (Retrieval), RepoBench-C (Code Completion), and RepoBench-P (Pipeline). Each task respectively measures the system's ability to retrieve the most relevant code snippets from other files as cross-file context, predict the next line of code with cross-file and in-file context, and handle complex tasks that require a combination of both retrieval and next-line prediction. RepoBench aims to facilitate a more complete comparison of performance and encouraging continuous improvement in auto-completion systems. RepoBench is publicly available at https://github.com/Leolty/repobench.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “…measure using broader APIs or general software engineering tasks, such as JuICe (Agashe et al., 2019), APIBench (Patil et al., 2023), RepoBench (Liu et al., 2023c), ODEX (Wang et al., 2022b), SWE-Bench (Jimenez et al., 2023), GoogleCodeRepo (Shrivastava et al., 2023), RepoEval (Zhang et al.,…”
LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023, methodology): “Note that this setting is different from the original setting of RepoBench (Liu et al., 2023), thus the results are not directly comparable to those reported in Liu et al. (2023).”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “After removing all files from these irrelevant folders, A GENTLESS divides each remaining file into chunks of code segments and computes the embedding for each chunk using an embedding model.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “Recently, AgentBench (Liu et al., 2023c) also mentions the challenge of LLM-as-Agent’s handling long interaction trajectories, but fails to incorporate it as a dedicated evaluation dimension.”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, methodology): “For all these tasks, we adhered to the standard metrics recommended by LongBench [2] (i.e., F1 for QA, Rouge-L for summarization, Acc. for synthetic and Edit Sim. for code generation.)”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, neutral): “…et al., 2025; Laban et al., 2024; Zhang et al., 2024b; Vodrahalli et al., 2024; Krishna et al., 2025), conversation (Bai et al., 2024a), coding (Liu et al., 2024a; Bogomolov et al., 2024), many-shot learning (Agarwal et al., 2024), and long-text generation (Bai et al., 2025; Wu et al., 2025b;…”
A Survey on Large Language Models for Code Generation (2024, neutral): “Issues for faster and lighter LLMs. Quantizationmethodtrade -offs.”
SweRank: Software Issue Localization with Code Ranking (2025, unknown): “Liu et al. (2024b) incorporate multi-task contrastive data that includes code contest generation (Billah et al., 2024), code summarization (Son-takke et al., 2022), code completion (Liu et al., 2024a), code translation (Pan et al., 2024) and code agent conversation (Jin et al., 2024).”
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis (2025, unknown): “…et al., 2021), and others (Li et al., 2022; Liu et al., 2023c; Li et al., 2023; Jain et al., 2024a; Banerjee et al., 2025; Ni et al., 2024b; Liu et al., 2023d; Du et al., 2023; Suresh et al., 2025b; Zhuo et al., 2024; Yin et al., 2022; Lai et al., 2023; Patil et al., 2025; Ugare et al.,…”