Incorporating External Knowledge through Pre-training for Natural Language to Code Generation

By Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, Graham Neubig. 2020

88 citations recorded by the source corpus.

Open-domain code generation aims to generate code in a general-purpose programming language (such as Python) from natural language (NL) intents. Motivated by the intuition that developers usually retrieve resources on the web when writing code, we explore the effectiveness of incorporating two varieties of external knowledge into NL-to-code generation: automatically mined NL-code pairs from the online programming QA forum StackOverflow and programming language API documentation. Our evaluations show that combining the two sources with data augmentation and retrieval-based data re-sampling improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa. The code and resources are available at https://github.com/neulab/external-knowledge-codegen.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

In-IDE Code Generation from Natural Language: Promise and Challenges (2021, methodology): “[115] (available online5), which is an improved version of the tree-based semantic parsing model by Yin and Neubig [121], further pre-trained on official API documentation in addition to the original training on Stack Overflow questions and answers.”
Grounded Copilot: How Programmers Interact with Code-Generating Models (2022, methodology): “In a similar vein, Xu et al. [2021] investigated the usefulness of an NL-to-code plugin previously developed by the same authors [Xu et al. 2020].”
Can we generate shellcodes via natural language? An empirical study (2021, methodology): “Ling et al. [56] and Yin and Neubig [88] proposed a novel neural architecture for code generation, while Xu et al. [84] incorporated pre-training and ﬁne-tuning of a model to generate Python snippets from natural language using the CoNaLa dataset [87].”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “[47] did a user study with 31 participants to evaluate the usefulness of a NL-to-code plugin [46].”
CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information (2022, methodology): “With the development of pre-training model [21], [22], [23] , it has also been proposed to help automatic code generation by pre-training models [24], [25], [26], [27].”
Natural Language to Code Translation with Execution (2022, background): “…text generation, there has been much work exploring natural language to code generation with end-to-end neural model architectures (Xiao et al., 2016; Ling et al., 2016; Rabinovich et al., 2017; Dong and Lapata, 2018; Suhr et al., 2018; Xu et al., 2020; Lachaux et al., 2021, inter alia ).”
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages (2022, methodology): “…Hu et al., 2018; Ahmad et al., 2020) and natural language (NL) to code generation (Ling et al., 2016; Rabinovich et al., 2017; Yin et al., 2018a; Xu et al., 2020; Norouzi et al., 2021; Wang et al., 2021), accompanied by code-speciﬁc tasks and benchmarks (Oda et al., 2015; Zhong et al., 2017;…”