LEVER: Learning to Verify Language-to-Code Generation with Execution

By Ansong Ni, Srini Iyer, Dragomir R. Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin. International Conference on Machine Learning, 2023

351 citations recorded by the source corpus.

The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the LLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/59fe7cb560651281cfc5db6b8940da0e3ba9dea6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

A Survey on Large Language Models for Code Generation (2024, neutral): “2 Overview: LLMs for Code Generations”
Explaining Competitive-Level Programming Solutions using LLMs (2023, background): “Li et al. (2022); Ni et al. (2023); Chen et al. (2022a) veriﬁed and selected candidate programs by running them on human-written or automatically generated test cases.”
Cognitive Architectures for Language Agents (2023, background): “Such digital grounding is cheaper and faster than physical or human interaction.”
MetaGPT: Meta Programming for Multi-Agent Collaborative Framework (2023, methodology): “Researchers trained classifiers to identify and reject erroneous programs [21], and developed the mechanism of iterative feedback to generate embedded control programs [22].”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “Additionally, LEVER [88] trains a verifier to distinguish and reject incorrect programs based on execution results.”
Teaching Large Language Models to Self-Debug (2023, methodology): “Following prior work evaluating large language models for text-to-SQL generation (Rajkumar et al., 2022; Shi et al., 2022; Ni et al., 2023), we evaluate SELF-DEBUGGING on the development set of the Spider benchmark (Yu et al.”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, positive): “Instead, to filter the experience, we leverage the environmental feedback that can remove critical failures; for instance, the program execution errors, retriever errors, and clearly wrong prefix of URL (Ni et al., 2023).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, neutral): “We first discuss the use of LLMs in the context of secure code programming [74] (or generation [62, 276, 190, 89]).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2022) ICL LLMs Scalar ✗ (rerank) Medium ✓ LEVER (Ni et al., 2023) SL Program Executor Scalar ✗ (rerank) High ✗ CodeRL (Le et al.”