Retrieval-Based Neural Code Generation

By Shirley Anugrah Hayati, R. Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, Graham Neubig. 2018

122 citations recorded by the source corpus.

In models to generate program source code from natural language, representing this code in a tree structure has been a common approach. However, existing methods often fail to generate complex code correctly due to a lack of ability to memorize large and complex structures. We introduce RECODE, a method based on subtree retrieval that makes it possible to explicitly reference existing code examples within a neural code generation model. First, we retrieve sentences that are similar to input sentences using a dynamic-programming-based sentence similarity scoring method. Next, we extract n-grams of action sequences that build the associated abstract syntax tree. Finally, we increase the probability of actions that cause the retrieved n-gram action subtree to be in the predicted code. We show that our approach improves the performance on two code generation tasks by up to +2.6 BLEU.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

SPoC: Search-based Pseudocode to Code (2019, background): “…text descriptions to a one-line program [42, 35, 43, 44, 21, 8], and while recent work explored generating longer programs from text descriptions [22, 40, 29, 15, 16, 14], these programs are mostly evaluated on syntactic metrics (e.g., exact match and BLEU score) rather than functional correctness.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, positive): “Recent work appearing after submission of this work supports this hypothesis by showing that augmenting AST-based models with AST subtrees retrieved via edit distance can boost the performance of AST-based models [15].”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “Such models are known to perform well in tasks with complex outputs such as program generation (Hayati et al., 2018; Hashimoto et al., 2018) and style transfer (Li et al., 2018).”
Non-Parametric Adaptation for Neural Machine Translation (2019, background): “Beyond NMT, there have been a few other attempts to incorporate non-parametric approaches into neural generative models (Guu et al., 2018; Hayati et al., 2018; Weston et al., 2018).”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, methodology): “…et al., 2018a), however, these approaches either use a heuristic way to exploit the retrieved logical form such as increasing the probability of actions (Hayati et al., 2018) or use a relevance function designed and learned based on expertise about the target logical form (Huang et al., 2018a).”
Semantic Scaffolds for Pseudocode-to-Code Generation (2020, background): “, 2016) that do not account for structure on the output side, or tree-based models (Allamanis et al., 2015; Rabinovich et al., 2017a; Yin and Neubig, 2017; Hayati et al., 2018; Iyer et al., 2019) that incorporate the syntax but not the semantics of the output domain.”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “And some use the retrievals to explicitly manipulate the token scores at each decoding step (Zhang et al., 2018; Hayati et al., 2018; Peng et al., 2019; Khandelwal et al., 2020, 2021).”
TreeGen: A Tree-Based Transformer Architecture for Code Generation (2019, methodology): “…have applied various neural architectures to this problem, such as sequence-to-sequence (Seq2Seq) models or sequence-to-tree (Seq2Tree) models (Sutskever, Vinyals, and Le 2014; Ling et al. 2016; Yin and Neubig 2017; Rabinovich, Stern, and Klein 2017; Hayati et al. 2018; Sun et al. 2019).”
In-IDE Code Generation from Natural Language: Promise and Challenges (2021, methodology): “, using hybrid models [38, 39] that may be able to combine the best of both worlds.”
Learning Programmatic Idioms for Scalable Semantic Parsing (2019, background): “We observe that Seq2Prod underperforms the Seq2Seq model of Iyer et al. (2017), most likely because a SQL query parse is much longer than the original query.”
Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021, methodology): “For the re-ranking models, fuzzy match score (Gu et al., 2018; Zhang et al., 2018; Hayati et al., 2018; Xu et al., 2020), attention mechanisms (Cao et al.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “In the retrieve process, fine-grained search for children nodes ultimately returns the parent node, effectively providing richer information.”
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, methodology): “In addition, there have been many retrieval-augmented methods for code generation [82– 88].”
CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information (2022, methodology): “In recent years, with the development of neural machine translation (NMT), sequence-to-sequence (Seq2Seq) model [5], [6], [7], [8] have been used for automatic code generation [9], [10], [11], [12].”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “…for a wide range of tasks such as question answering (Chen et al., 2017; Lee et al., 2019; Nakano et al., 2021), machine translation (Zhang et al., 2018), code generation (Hayati et al., 2018), language modeling (Khandelwal et al., 2020), and other knowledge-intensive tasks (Lewis et al., 2020).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “, 2018), code generation (Hayati et al., 2018), language modeling (Khandelwal et al.”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “of existing code and can generate new code given some forms of specifications such as natural language descriptions or incomplete code [5, 16, 17, 22, 38, 39, 48, 49].”
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions (2019, methodology): “Our work is also related to recently proposed approaches to code generation by editing (Hayati et al., 2018; Yin et al., 2019; Hashimoto et al., 2018).”