L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

By Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong and 4 more. 2023

44 citations recorded by the source corpus.

Abstract Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs. Despite promising results, there is a notable lack of a comprehensive evaluation of these models’ language-to-code generation capabilities. Existing studies often focus on specific tasks, model architectures, or learning paradigms, leading to a fragmented understanding of the overall landscape. In this work, we present L2CEval, a systematic evaluation of the language-to-code generation capabilities of LLMs on 7 tasks across the domain spectrum of semantic parsing, math reasoning, and Python programming, analyzing the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition, we assess confidence calibration, and conduct human evaluations to identify typical failures across different tasks and models. L2CEval offers a comprehensive understanding of the capabilities and limitations of LLMs in language-to-code generation. We release the evaluation framework1 and all model outputs, hoping to lay the groundwork for further future research. All future evaluations (e.g., LLaMA-3, StarCoder2, etc) will be updated on the project website: https://l2c-eval.github.io/.

Subfields: Artificial Intelligence, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “These primarily focus on natural language to Python code generation: HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2023b), APPS (Hendrycks et al., 2021), Code-Contests (Li et al., 2022), MBPP (Austin et al., 2021), L2CEval (Ni et al., 2023).”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, unknown): “To obtain the model performance and predictions on each of the dataset examples, we use the evaluation framework and model outputs from L2CEval (Ni et al., 2023).”
A Survey on Large Language Models for Code Generation (2024, unknown): “Finally, we explore the significant potential of LLMs in many different coding tasks, including code generation, completion, search, debugging, and translation, which significantly boost productivity and efficiency for users in writing code.”
CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis (2025, unknown): “…2023b), APPS (Hendrycks et al., 2021), and others (Li et al., 2022; Liu et al., 2023c; Li et al., 2023; Jain et al., 2024a; Banerjee et al., 2025; Ni et al., 2024b; Liu et al., 2023d; Du et al., 2023; Suresh et al., 2025b; Zhuo et al., 2024; Yin et al., 2022; Lai et al., 2023; Patil et al.,…”