Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

By Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, Graham Neubig. 2018

366 citations recorded by the source corpus.

For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. StackOverflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high-quality code snippets. However, existing heuristic methods (e.g. pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “, input-output examples [12, 28, 19, 13], partial implementations [58], or natural language descriptions [74, 73, 72, 42, 18] of the desired program as input.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “There are a number of existing single-language NL2Code benchmarks [8], [9], [10].”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack-Overﬂow posts (Yin et al., 2018; Wang et al., 2022), but face challenges in enabling execution (Lai et al., 2022; Chandel et al., 2022).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Notably, such a check is different from the standard surface-form metrics such as CodeBLEU (Ren et al., 2020), which requires the whole model prediction to be uniformly similar to a reference solution; instead, DS-1000 precisely targets small but important parts of surface form.”
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018, background): “, 2018), and CoNaLa (Yin et al., 2018).”
Multi-lingual Evaluation of Code Generation Models (2022, neutral): “Various existing benchmarks for code generation are either match-based (Lu et al., 2021; Yin et al., 2018; Wang et al., 2022; Husain et al., 2019; Clement et al., 2021) or focused mostly on Python (Chen et al.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “Yin et al. [90] mine pairs of natural language and code from Stack Overﬂow to support tasks such as code synthesis from natural language.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, background): “There are a number of existing single-language NL2Code benchmarks [9, 10, 11].”
Natural Language to Code Translation with Execution (2022, background): “Multiple benchmarks have been proposed to evaluate code model performance (Miceli Barone and Sennrich, 2017; Yin et al., 2018; Hendrycks et al., 2021; Lu et al., 2021, inter alia ).”
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020, methodology): “We use the same evaluation metric as the CoNaLa benchmark, corpus-level BLEU calculated on target code outputs in test set.”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, neutral): “Several code generation benchmarks are collected from raw data from GitHub and StackOverflow, and involve professional annotators to enhance the quality of the data (Iyer et al., 2018; Agashe et al., 2019; Yin et al., 2018).”
Out of the BLEU: how should we assess quality of the Code Generation models? (2022, methodology): “[39] and consists of 2,879 examples (split into 2,379 training and 500 test examples), crawled from Stack Overflow and then manually curated by human annotators.”
A Neural Model for Generating Natural Language Summaries of Program Subroutines (2019, neutral): “[21], Movshovitz et al.”
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages (2022, methodology): “We train baseline models in their available settings, then tokenize the generated and reference code snippets following Yin and Neubig (2018) to evaluate the BLEU-4 scores.”
CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021, background): “A number of open-sourced datasets with a large amount of text-code pairs have been proposed for the purposes of code search (Husain et al., 2019; Gu et al., 2018; Nie et al., 2016) and code question answering (Yao et al., 2018; Yin et al., 2018; Heyman and Cutsem, 2020).”
Program Synthesis and Semantic Parsing with Learned Code Idioms (2019, background): “In the last decade, it has advanced dramatically thanks to the novel neural and neuro-symbolic techniques [5, 10, 19], ﬁrst mass-market applications [28], and massive datasets [9, 39, 41].”
A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages (2022, methodology): “There are a number of single-language NL2Code benchmarks (Yin et al. 2018; Kulal et al. 2019; Hendrycks et al. 2021).”