Multi-lingual Evaluation of Code Generation Models

By Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang and 15 more. 2022

247 citations recorded by the source corpus.

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

Citations in the corpus

SantaCoder: don't reach for the stars! (2023, methodology): “Some groups have released model weights (Fried et al., 2022; Nijkamp et al., 2022) or provided access to the model through a paid API service (Chen et al., 2021; Athiwaratkun et al., 2022), but these works did not release the full training data or the preprocessing methods that were used.”
The Stack: 3 TB of permissively licensed source code (2022, background): “Recently, these benchmarks have been extended from Python to other programming languages (Athiwaratkun et al., 2022; Cassano et al., 2022).”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “Their variants have been proposed to cover more languages, (Wang et al., 2022a; Zheng et al., 2023; Cassano et al., 2022; Athiwaratkun et al., 2022).”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “(Chen et al., 2021) 1 1,325 MBPP (Austin et al., 2021) 1 1,500 TransCoder-ST (Roziere et al., 2021b) 3 ∞ APPS(Hendrycks et al., 2021) 1 22,711 MBXP (Athiwaratkun et al., 2022) 10 1,500 CodeContests (Li et al., 2022) 3 Our contributions in this paper are summarized as follows : 1.”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, methodology): “Athiwaratkun et al. (2023) developed the MBXP dataset using a scalable transformation framework to generate a dataset that spans ten programming languages.”
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, background): “To date, many code generation benchmarks (Yin et al., 2018; Athiwaratkun et al., 2022; Hendrycks et al., 2021) focus on generating independent programs.”
Measuring The Impact Of Programming Language Distribution (2023, result): “While MBXP is quite similar to BabelCode, it is not open-sourced and requires that the input benchmarks be in Python.”
Structured Chain-of-Thought Prompting for Code Generation (2023, unknown): “Specifically, we generate n ≥ k programs per requirement (in this paper, we use n = 20, k ∈ [1, 3, 5]), count the number of solved requirements c , and calculate the unbiased Pass@k :”
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023, background): “…Metrics To alleviate previous issues, execution-based evaluation counts a generated code snippet as correct if it produces the required outputs when run with given inputs (Chen et al., 2021; Athiwaratkun et al., 2022; Li et al., 2022; Wang et al., 2022; Lai et al., 2022; Huang et al., 2022).”
Evaluating Large Language Models in Class-Level Code Generation (2024, methodology): “…sources: (i) Top-10 popular datasets with the highest download volumes from Huggingface code generation datasets [3], (ii) benchmarks associated with recent LLM papers (released between June 2021 and June 2023), and (iii) enhanced benchmarks such as Hu-manEval+ [44] and Multi-HumanEval [14].”
A Survey on Large Language Models for Code Generation (2024, unknown): “These tasks represent various aspects of code generation, including foundational tasks like code generation and code completion, to advanced tasks such as advanced code generation and code search, and auxiliary tasks including de-bugging and code translation.”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “However, some large language models can adapt and learn new languages in context, allowing them to generate code in languages not in their training data (Athiwaratkun et al., 2022).”
Teaching Large Language Models to Self-Debug (2023, background): “def count_ways(n): A = [0] * (n + 1) B = [0] * (n + 1) A[0] = 1 A[1] = 0”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, unknown): “MBXP [43] is a concurrenteffort by Amazon Research to evaluate code generation models.”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, methodology): “From several existing benchmarks (Wang et al., 2022; Athiwaratkun et al., 2022; Chen et al., 2021; Liu et al., 2023; Gu et al., 2024; Zheng et al., 2023; Cassano et al., 2022; Jimenez et al., 2023; Du et al., 2023; Odena et al., 2021; Puri et al., 2021; Ahmad et al., 2021), we chose the programs in…”