StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

By Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, Huan Sun. 2018

113 citations recorded by the source corpus.

Stack Overflow (SO) has been a great source of natural language questions and their code solutions (i.e., question-code pairs), which are critical for many tasks including code retrieval and annotation. In most existing research, question-code pairs were collected heuristically and tend to have low quality. In this paper, we investigate a new problem of systematically mining question-code pairs from Stack Overflow (in contrast to heuristically collecting them). It is formulated as predicting whether or not a code snippet is a standalone solution to a question. We propose a novel Bi-View Hierarchical Neural Network which can capture both the programming content and the textual context of a code snippet (i.e., two views) to make a prediction. On two manually annotated datasets in Python and SQL domain, our framework substantially outperforms heuristic methods with at least 15% higher F1 and accuracy. Furthermore, we present StaQC (Stack Overflow Question-Code pairs), the largest dataset to date of ~148K Python and ~120K SQL question-code pairs, automatically mined from SO using our framework. Under various case studies, we demonstrate that StaQC can greatly help develop data-hungry models for associating natural language with programming language

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “To shrink the candidates to be annotated for each query, we select the top two functions with the highest query-code similarity computed by a CodeBERT-based code retrieval model, which is trained on 148K automated-minded Python Stack Overflow Question-Code (StaQC) [92] with the default parameters pr”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, methodology): “To ensure that our query set is representative, we obtained common search queries from Bing that had high click-through rates to code and combined these with intent rewrites in StaQC [24].”
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, methodology): “s. We can view these methods from several perspectives: Data Sources: First, what data sources do they use to mine their data? Our work falls in the line of mining intent-snippet pairs from SO (e.g., [15, 44, 46, 48]), while there has been research on miningfromotherdatasourcessuchasAPIdocumentation”
NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System (2018, background): “Extracting good alignments from them is very challenging (Quirk et al., 2015; Iyer et al., 2016; Yao et al., 2018).”
A Neural Model for Generating Natural Language Summaries of Program Subroutines (2019, background): “a neural model for source code search, another task growing in popularity and facilitated by public datasets [34].”
CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021, methodology): “A number of open-sourced datasets with a large amount of text-code pairs have been proposed for the purposes of code search (Husain et al., 2019; Gu et al., 2018; Nie et al., 2016) and code question answering (Yao et al., 2018; Yin et al., 2018; Heyman and Cutsem, 2020).”
CodeTrans: Towards Cracking the Language of Silicone's Code Through Self-Supervised Deep Learning and High Performance Computing (2021, methodology): “StaQC: For the SQL unlabeled dataset, we chose StaQC14 (Yao et al., 2018).”
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020, background): “However, recently there has been a move towards applying semantic parsing to automatically generating source code in general-purpose programming languages (Yin et al., 2018; Yao et al., 2018; Lin et al., 2018; Agashe et al., 2019; Yao et al., 2019).”
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning (2019, methodology): “, those collected from Stack Overflow [51] by [21, 63], we define the two tasks as:”
A Survey on Semantic Parsing (2018, background): “In Zettlemoyer and Collins [2005], the lexicon is induced as part of the learning procedure by only preserving lexical entries that result in the required parse. However such a system would have issues due to the inflexibility of CCG when it is applied to raw natural language inputs, having variable”
CLARC: C/C++ Benchmark for Robust Code Search (2026, unknown): “Third, code search benchmarks (Liu et al., 2024a; Khan et al., 2024) rarely evaluate model robustness against textual perturbations, such as variable renaming Li et al., 2025; 2024; Yao et al., 2018; Heyman and Cutsem, 2020; Yin et al., 2018).”