MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages

By Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, Graham Neubig. 2022

63 citations recorded by the source corpus.

While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-Code pairs in three languages: Spanish, Japanese, and Russian. We present a systematic evaluation on MCoNaLa by testing state-of-the-art code generation systems. Although the difficulties vary across three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.

Subfields: Information Systems, Software, Evaluation & Benchmarks, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

In Signal Processing

Citations in the corpus

MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “We note that MCoNaLA [36], which extends CoNaLa to Spanish, Japanese, and Russian, is the only currently available benchmark for evaluating code generation from multiple natural languages.”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “Some datasets preserve natural coverage from interactive Jupyter Notebooks (Agashe et al., 2019) or Stack-Overﬂow posts (Yin et al., 2018; Wang et al., 2022), but face challenges in enabling execution (Lai et al., 2022; Chandel et al., 2022).”
Multi-lingual Evaluation of Code Generation Models (2022, neutral): “Various existing benchmarks for code generation are either match-based (Lu et al., 2021; Yin et al., 2018; Wang et al., 2022; Husain et al., 2019; Clement et al., 2021) or focused mostly on Python (Chen et al.”
Measuring The Impact Of Programming Language Distribution (2023, background): “Related Works Code Evaluation Existing code benchmarks have primarily focused on surface matching evaluation (Lu et al., 2021; Yin et al., 2018; Wang et al., 2022b; Husain et al., 2019).”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “In addition, MoCoNaLa (Wang et al., 2022a) extends CoNaLa to support more natural languages.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, background): “We note that MCoNaLA [37], which extends CoNaLa to Spanish, Japanese, and Russian, is the only currently available benchmark for evaluating code generation from multiple natural languages.”