Synchromesh: Reliable code generation from pre-trained language models

By Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, Sumit Gulwani. 2022

213 citations recorded by the source corpus.

Large pre-trained language models have been used to generate code,providing a flexible interface for synthesizing programs from natural language specifications. However, they often violate syntactic and semantic rules of their output language, limiting their practical usability. In this paper, we propose Synchromesh: a framework for substantially improving the reliability of pre-trained models for code generation. Synchromesh comprises two components. First, it retrieves few-shot examples from a training bank using Target Similarity Tuning (TST), a novel method for semantic example selection. TST learns to recognize utterances that describe similar target programs despite differences in surface natural language features. Then, Synchromesh feeds the examples to a pre-trained language model and samples programs using Constrained Semantic Decoding (CSD): a general framework for constraining the output to a set of valid programs in the target language. CSD leverages constraints on partial outputs to sample complete correct programs, and needs neither re-training nor fine-tuning of the language model. We evaluate our methods by synthesizing code from natural language descriptions using GPT-3 and Codex in three real-world languages: SQL queries, Vega-Lite visualizations and SMCalFlow programs. These domains showcase rich constraints that CSD is able to enforce, including syntax, scope, typing rules, and contextual logic. We observe substantial complementary gains from CSD and TST in prediction accuracy and in effectively preventing run-time errors.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

Citations in the corpus

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, neutral): “This approach bears resemblance to several recent works (Poesia et al., 2022; Gao et al., 2020; Liu et al., 2021; Rubin et al., 2021).”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “, 2022), and additional techniques at inference time further improve the performance (Poesia et al., 2022; Shi et al., 2022).”
Jigsaw: Large Language Models meet Program Synthesis (2021, methodology): “Syncromesh [31] uses build context prompts for PTLMs similar to how Jigsaw does in pre-processing.”
Few-Shot Semantic Parsing with Language Models Trained on Code (2021, methodology): “When W = 1 and T = 0, which is equivalent to Synchromesh’s approach, we obtain very similar results to constrained greedy decoding (beam size 1).”
Repairing Bugs in Python Assignments Using Large Language Models (2022, background): “Existing work has also observed similar phenomena in the related area of natural language to code generation [26].”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “In the retrieve process, fine-grained search for children nodes ultimately returns the parent node, effectively providing richer information.”
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (2022, methodology): “Recent attempts employ deep learning methods, specifically Transformer-based pretrained language models (LMs) Vaswani et al. [2017], Brown et al.”
Large Language Model Alignment: A Survey (2023, unknown): “, text-to-code generation (Poesia et al., 2022), planning (Huang et al.”
Language Models of Code are Few-Shot Commonsense Learners (2022, methodology): “Popular techniques typically train a retriever, which is used to fetch the examples in the training set that are closest to T test (Liu et al., 2021; Rubin et al., 2021; Poesia et al., 2021).”
Learning Performance-Improving Code Edits (2023, methodology): “Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
CRANE: Reasoning with constrained LLM generation (2025, unknown): “However, as highlighted in recent studies (Ugare et al., 2024b; Lundberg et al., 2023; Poesia et al., 2022), pre-trained LLM outputs do not always comply with downstream tools’ input requirements.”