CoSQA: 20,000+ Web Queries for Code Search and Question Answering

By Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, Nan Duan. 2021

154 citations recorded by the source corpus.

Finding codes given natural language query is beneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce CoSQA dataset. It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance text-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that, evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1% and incorporating CoCLR brings a further improvement of 10.5%.

Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Question Answering & Reading Comprehension

In Code Generation & Program Synthesis

Citations in the corpus

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “Since there lacks a direct training and validation set, we use the CoSQA [31] as the training resources with 20,604 pairs of query and code.”
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “Further, due to limited test data and context size, we considered only 1-Shot prompts which could be extended to multi-shot in future.”
UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022, methodology): “We conduct experiments on three datasets, namely CSN (Guo et al., 2020), AdvTest (Lu et al., 2021) and CosQA (Huang et al., 2021).”
CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “We demonstrate that contrastive learning using our dataset leads to state-of-the-art code embedding models, outperforming even larger models (Zhang et al., 2024; Wang et al., 2023b) on a variety of code retrieval tasks (Husain et al., 2019; Huang et al., 2021; Lu et al., 2021).”
CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding (2026, unknown): “…retrieval through APPS, CoSQA, SyntheticText2SQL, CLARC, Code-SearchNet, CodeTransOcean-DL, CodeFeedback-ST, and CPRet (Hendrycks et al., 2021; Huang et al., 2021; Muennighoff et al., 2023; Wang et al., 2026; Husain et al., 2019; Yan et al., 2023; Zheng et al., 2024; Deng et al., 2025).”
CLARC: C/C++ Benchmark for Robust Code Search (2026, unknown): “First, most current datasets (Huang et al., 2021; Husain et al., 2020; Li et al., 2025; Lu et al., 2021; Li et al., 2024; Yao et al., 2018; Heyman and Cutsem, 2020; Yin et al., 2018) prioritize research-favored languages like Python, while neglecting text-to-code tasks of systems programming…”
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2025, unknown): “These benchmarks largely overlook critical, code-specific quality dimensions such as correctness , efficiency , security , and maintainability [6, 7, 8, 9].”
CoRet: Improved Retriever for Code Editing (2025, unknown): “Benchmarks for code retrieval The scope of standard benchmarks for semantic matching or code search between a natural language query and code such as CoSQA (Huang et al., 2021), CodeSearchNet (Husain et al., 2019), AdvTest (Lu et al., 2021), CodeXGLUE (Lu et al., 2021) are based on docstrings and co”