CodeRAG-Bench: Can Retrieval Augment Code Generation?

By Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yi-Qing Xie, Graham Neubig, Daniel Fried. 2024

Subfields: Retrieval-Augmented Generation, Evaluation & Benchmarks, Code Generation & Program Synthesis

Citations in the corpus

• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “Code retrieval is crucial for advancing Retrieval-Augmented Code Generation (RACG) (Wang et al., 2024b) with large language models (LLMs), where providing contextual examples significantly improves the relevance and accuracy of generated code.”
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (2025, unknown): “En-1 dowing RAG with this capability is therefore especially promising for personalized knowledge collections, such as long-context question answering (Bai et al., 2023), code generation (Wang et al., 2024b; Miao et al., 2024), and AI assistants over personal projects (Martin and Johnson, 2023).”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “…language modeling (Khandelwal et al., 2020; Borgeaud et al., 2022) to applications in agents (Wu et al., 2023; Wang et al., 2024a), code generation (Wang et al., 2024c; Zhang et al., 2023a), robotics (Anwar et al., 2024), medicine (Jeong et al., 2024), and protein research (Jumper et al., 2021),…”
• Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation (2025, unknown): “For instance, CodeRAG-Bench [19] establishes a comprehensive evaluation framework for Python code RACG; CodeGRAG [55] explore Python and C++ RACG which use graphical view of code blocks; RRG [20] introduces a code refactorer module in Python and Java RACG.”
• FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “Finally, we keep questions with an accepted answer , prioritizing precision over quantity in retaining questions with only high-quality answers accepted and upvoted in the Stack Overflow community.”
• LocAgent: Graph-Guided LLM Agents for Code Localization (2025, unknown): “Poor code localization leads to incomplete fixes, introduces new bugs, and significantly extends development cycles.”
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval (2024, unknown): “Code-RAG systems (Parvez et al., 2021; Liu et al.; Jimenez et al., 2024; Wang et al., 2024) also leverage code retrieval to minimize hallucinations in generated code from LLMs, ensuring more accurate outputs.”
• RepoNav: From Snippet Retrieval to File-Centered Repository Navigation for Code Agents (2026, unknown): “…generation (Lewis et al., 2020) has been widely adopted for injecting external knowledge into language models; in the code domain, retrieval-based meth-ods such as RepoCoder (Zhang et al., 2023) and CodeRAG-Bench (Wang et al., 2025b) extend this paradigm to repository-level tasks.”
• LLMs as Packagers of HPC Software (2025, unknown): “Our embedding-based method follows the canonical retrieve-then-generate paradigm commonly used in code generation settings [22, 33], often referred to as retrieval-augmented generation (RAG).”
• Improving Code Localization with Repository Memory (2025, unknown): “Retrieval-based methods represent the most conventional approach, leveraging lexical or semantic matching to rank code snippets based on their proximity to the query (Wang et al., 2025b).”
• AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming (2025, unknown): “However, Wang et al. [2025] has found that retrievers struggle to fetch relevant and helpful context for code generation.”
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2025, unknown): “Code retrieval is crucial for code generation by introducing relevant code snippets as references for large language model (LLM) generation, significantly advancing automated software development [1, 2, 3, 4, 5].”
• CoRet: Improved Retriever for Code Editing (2025, unknown): “The repository-level retrieval benchmarks such as Re-poEval (Zhang et al., 2023a), Repobench (Liu et al., 2024), CodeRAG-Bench (Wang et al., 2024), EvoCodeBench (Li et al., 2024a) are typically used for code completion tasks where they do not have natural language queries or the retrieval is only…”