Natural Language to Code Translation with Execution

By Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke S. Zettlemoyer, Sida I. Wang. 2022

181 citations recorded by the source corpus.

Generative models of code, pretrained on large corpora of programs, have shown great success in translating natural language to code (Chen et al., 2021; Austin et al., 2021; Li et al., 2022, inter alia). While these models do not explicitly incorporate program semantics (i.e., execution results) during training, they are able to generate correct solutions for many problems. However, choosing a single correct program from a generated set for each problem remains challenging. In this work, we introduce execution result–based minimum Bayes risk decoding (MBR-EXEC) for program selection and show that it improves the few-shot performance of pretrained code models on natural-language-to-code tasks. We select output programs from a generated candidate set by marginalizing over program implementations that share the same semantics. Because exact equivalence is intractable, we execute each program on a small number of test inputs to approximate semantic equivalence. Across datasets, execution or simulated execution significantly outperforms the methods that do not involve program semantics. We find that MBR-EXEC consistently improves over all execution-unaware selection methods, suggesting it as an effective approach for natural language to code translation.

Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “, 2022), and additional techniques at inference time further improve the performance (Poesia et al., 2022; Shi et al., 2022).”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “…help cross-lingual transfer on a wide range of NLP tasks such as cross-lingual named entity recognition (Pires et al., 2019; Mulcaire et al., 2019), zero-shot cross-lingual dependency parsing (Schuster et al., 2019; Shi et al., 2022b), and bilingual lexicon induction (Shi et al., 2021).”
CodeT: Code Generation with Generated Tests (2022, methodology): “Moreover, some work has been proposed to leverage the execution information (Shi et al., 2022; Li et al., 2022b; Le et al., 2022; Lahiri et al., 2022).”
Teaching Large Language Models to Self-Debug (2023, methodology): “Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
Self-Edit: Fault-Aware Code Editor for Code Generation (2023, methodology): “Recently some approaches are also proposed to post-process programs generated by LLMs (Shi et al., 2022; Inala et al., 2022; Chen et al., 2022; Zhang et al., 2022).”
Demystifying GPT Self-Repair for Code Generation (2023, methodology): “This literature has predominantly focused on evaluating models in terms of either raw accuracy or the pass@k metric [5, 17], often leveraging filtering techniques based on execution [21, 31] or ranking [5, 15, 37] to reduce the number of samples which are considered for the final answer.”
LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, methodology): “Previous Work without Finetuning MBR-Exec (Shi et al., 2022) 63.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “The third is the consistency-based method (Wang et al., 2022; Shi et al., 2022; Zhao et al., 2023a).”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, methodology): “Including test data in prompts has been a known practice to improve the performance of models in programming tasks (Chen et al., 2022; Zhong et al., 2022; Shi et al., 2022; Zhang et al., 2023).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, background): “For other tasks, having a better pass@k also provides opportunities for post-generation reranking meth-ods such as (Shi et al., 2022; Zhang et al., 2023; Ni et al., 2023b).”
INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (2024, background): “Among the recent advancements, prompting LLMs to generate multiple responses to the same question and evaluating the self-consistency of those responses has been proven effective in hallucination detection (Wang et al., 2022; Shi et al., 2022).”
Binding Language Models in Symbolic Languages (2022, methodology): “Finally, the output answer A is derived via a majority voting strategy over the set of produced answers A (the voting method is similar to the one used by MBR-EXEC (Shi et al., 2022), which we elaborate on in Appendix A.3).”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, methodology): “Self-consistency (SC; Wang et al., 2022c; Shi et al., 2022a) may further boost the reasoning performance by marginal-izing over intermediate reasoning steps that share the same ﬁnal result.”
SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (2024, unknown): “For coding tasks, execution-based approaches have been shown to improve code generation performance [18, 24].”
Compositional Semantic Parsing with Large Language Models (2022, methodology): “temperature-based decoding to sample multiple predictions per list, and finally aggregate predictions by plurality vote using self-consistency (Wang et al., 2022; Shi et al., 2022; Li et al., 2022).”
FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, background): “, 2022), and execution-based evaluation for code (Shi et al., 2022).”