XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence

By Ming Zhu, Aneesh Jain, Karthik Suresh, R. Ravindran, Sindhu Tipirneni, Chandan K. Reddy. 2022

105 citations recorded by the source corpus.

Recent advances in machine learning have significantly improved the understanding of source code data and achieved good performance on a number of downstream tasks. Open source repositories like GitHub enable this process with rich unlabeled code data. However, the lack of high quality labeled data has largely hindered the progress of several code related tasks, such as program translation, summarization, synthesis, and code search. This paper introduces XLCoST, Cross-Lingual Code SnippeT dataset, a new benchmark dataset for cross-lingual code intelligence. Our dataset contains fine-grained parallel data from 8 languages (7 commonly used programming languages and English), and supports 10 cross-lingual code tasks. To the best of our knowledge, it is the largest parallel dataset for source code both in terms of size and the number of languages. We also provide the performance of several state-of-the-art baseline models for each task. We believe this new dataset can be a valuable asset for the research community and facilitate the development and validation of new methods for cross-lingual code intelligence.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis, Signal Processing, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Yuxiao Dong
3. Zhilin Yang
4. Xu Zou
5. Yufei Xue
6. Xiao Xia
7. Andi Wang
8. Yang Li

In Information Systems

1. Yuxiao Dong
2. Jie Tang
3. Xu Zou

In Code Generation & Program Synthesis

1. Yuxiao Dong
2. Zhilin Yang
3. Jie Tang
4. Yufei Xue
5. Xiao Xia
6. Andi Wang
7. Yang Li
8. Zihan Wang

In Signal Processing

1. Yuxiao Dong
2. Zhilin Yang
3. Jie Tang
4. Xu Zou

Citations in the corpus

• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “Specifically, we use the unbiased method to estimate pass@ 𝑘 [7]: where 𝑛 is the total number of generations ( 𝑛 =200 in this work), 𝑘 is the sampling budget (typically 𝑘 ∈ { 1, 10, 100 } ), and 𝑐 is the number of samples that pass all test cases.”
• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “Thirdly and more importantly, most studies (e.g., widely used benchmarks CodeXGLUE (Lu et al., 2021) and XLCoST (Zhu et al., 2022)) rely only on matching-based evaluation metrics and overlook the practical applicability of the generated code, even though the ultimate goal of the generated code is…”
• Lost in Translation, Found in Summary: Summary-Driven Supervision for Cross-Language Code Retrieval (2026, unknown): “(1) We leverage an empirical study benchmark combining CodeJamData [35], AtCoder [35], XL-CoST [75], and CodeSearchNet [15].”