CoIR: A Comprehensive Benchmark for Code Information Retrieval Models

By Xiang-Yang Li, Kui-Cai Dong, Yi Lee, Wei Xia, Yi-Chun Yin, Hao Zhang, Yong Liu, Yasheng Wang, Rui-Ming Tang. 2024

Subfields: Information Retrieval & Search, Evaluation & Benchmarks, Code Generation & Program Synthesis

Citations in the corpus

• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “Finally, using our improved code retriever and reranker together, we show significant improvements in function localization (Xia et al., 2024), a key aspect in addressing real-world software engineering challenges (Jimenez et al., 2024).”
• HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions (2026, unknown): “• Code: NanoCoIR (Li et al., 2024), NanoCodeRAG (Wang et al., 2025).”
• Granite Embedding Multilingual R2 Models (2026, unknown): “…We evaluate on the MTEB-v2 Retrieval benchmark (Enevoldsen et al., 2025) • Code Retrieval: We evaluate on code retrieval tasks of the MTEB-Code benchmark (Enevoldsen et al., 2025), including tasks from COIR (Li et al., 2024), which consists of text-to-code, code-to-text, and hybrid code retrieval.”
• Granite Embedding R2 Models (2025, unknown): “We also include evaluation on the popular BEIR benchmark (Thakur et al., 2021) in Appendix D. • Code Retrieval: We evaluate on code retrieval tasks of the COIR benchmark (Li et al., 2024), which consists of text-to-code, code-to-text, and hybrid code retrieval.”
• SweRank: Software Issue Localization with Code Ranking (2025, unknown): “These typically include query-to-code retrieval (Li et al., 2024b), which aims to find code implementing a described functionality, and code-to-code retrieval (Wang et al., 2023a; Li et al., 2024a), focused on identifying semantically similar code fragments.”
• Granite Embedding Models (2025, unknown): “We evaluate the model’s code retrieval capability on the COIR benchmark Li et al. (2024), consisting of 10 datasets across 7 domains in Table 6.”
• Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval (2026, unknown): “Future VDR research must address implicit retrieval intents , where queries involve fuzzy constraints that can only be resolved through world-knowledge synthesis (Zhang et al., 2025g; Wei et al., 2024b).”
• FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “Finally, we keep questions with an accepted answer , prioritizing precision over quantity in retaining questions with only high-quality answers accepted and upvoted in the Stack Overflow community.”
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval (2024, unknown): “To understand the base model’s impact, we examine: (1) if training from a text retrieval model out-performs a generation model, and (2) if a code-specific generation model offers more advantages than a general language model.”
• Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval (2026, unknown): “Standard text-and code-retrieval benchmarks (Thakur et al., 2021; Muennighoff et al., 2023; Li et al., 2025; Geng et al., 2026) evaluate zero-shot transfer but do not cover schema retrieval, a setting in which we find that even strong general-purpose and code embed-ders underperform.”
• CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding (2026, unknown): “CodeSearchNet (Husain et al., 2019) and many code tasks in MTEB (Muennighoff et al., 2023) and CoIR (Li et al., 2025) mainly evaluate de-contextualized snippet retrieval under fixed corpora and simplified query-document mappings.”
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2025, unknown): “More recently, CoIR [9] unified ten existing datasets and eight retrieval tasks into a comprehensive benchmark, enabling holistic evaluation across different programming domains and query modalities.”
• CoRet: Improved Retriever for Code Editing (2025, unknown): “CoIR (Li et al., 2024b) is a recent benchmark that curated a collection of multiple datasets for various scenarios, including text-to-code, code-to-text, code-to-code retrieval, thus going beyond our target setting.”