MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation

By Federico Cassano, John Gouwar, Daniel Nguyen, S. Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman and 3 more. 2023

282 citations recorded by the source corpus.

Large language models have demonstrated the ability to generate both natural language and programming language text. Although contemporary code generation models are trained on corpora with several programming languages, they are tested using benchmarks that are typically monolingual. The most widely used code generation benchmarks only target Python, so there is little quantitative evidence of how code generation models perform on other programming languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark (Chen et al., 2021) and MBPP benchmark (Austin et al., 2021) to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex (Chen et al., 2021), CodeGen (Nijkamp et al., 2022) and InCoder (Fried et al., 2022). We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, methodology): “For that, we use the MultiPL-E benchmark (Cassano et al., 2023).”
StarCoder: may the source be with you! (2023, methodology): “Out of the languages present in MultiPL-E (Cassano et al., 2023), only D and Swift were not included in the training set.”
Competition-Level Problems are Effective LLM Evaluators (2023, methodology): “Unlike simpler tasks on code, such as HumanEval (Chen et al., 2021a), MBPP (Austin et al., 2021), MultiPL-E (Cassano et al., 2023), competition-level programming problems necessitate an advanced understanding of data structures, algorithms, and problem-solving techniques.”
Agentless: Demystifying LLM-based Software Engineering Agents (2024, background): “However, instead of embedding all files in the repository, A GENTLESS first filters out irrelevant folders.”
DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “To evaluate the model’s multilingual capabilities, we expanded the Python problems of Humaneval Benchmark to seven additional commonly used programming languages, namely C++, Java, PHP, TypeScript (TS), C#, Bash, and JavaScript (JS) (Cassano et al., 2023).”
Qwen2 Technical Report (2024, unknown): “…al., 2018) (25-shot), HumanEval (Chen et al., 2021) (0-shot), MBPP (Austin et al., 2021) (0-shot), EvalPlus(Liu et al., 2023a) (0-shot), MultiPL-E (Cassano et al., 2023) (0-shot on Python, C++, Java, PHP, Type-Script, C#, Bash, and JavaScript), GSM8K (Cobbe et al., 2021) (5-shot), MATH (Hendrycks…”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “X [76] (C++, Javascript and Go), CodeContests [33] (C++ and Java) and MultiPL-E [9] (extending H UMAN E VAL and MBPP to 18 programming languages).”
Qwen2.5-Omni Technical Report (2025, unknown): “Specifically, we utilize MMLU-Pro (Wang et al., 2024f), MMLU-redux (Gema et al., 2024) and Livebench0803 (White et al., 2024) for general evaluation, GPQA (Rein et al., 2023), GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) for mathematics & science, HumanEval (Chen et al., 2021), MBPP”
Kimi K2: Open Agentic Intelligence (2025, unknown): “For coding, we adopt LiveCodeBench v6 [32](questions from August 2024 to May 2025), OJBench [78], MultiPL-E [6], SWE-bench Verified [33, 85], TerminalBench [72], Multi-SWE-bench [87], SWE-Lancer [51], PaperBench [66], and Aider-Polyglot [17].”
Qwen3 Technical Report (2025, unknown): “…et al., 2023a) (0-shot) (Average of HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), Humaneval+, MBPP+) (Liu et al., 2023a), MultiPL-E (Cassano et al., 2023) (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript), MBPP-3shot (Austin et al., 2021), CRUX-O of CRUXEval…”
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation (2025, unknown): “HumanEval-X [42], MultiPL-E [52] and CruxEval-X [56] enable parallel evaluation across multiple PL s by translating existing problems, and McEval [44] further enhances the diversity of evaluation data.”