xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

By Mohammad Abdullah Matin Khan, M Saiful Bari, Do Xuan Long, Weishi Wang, Md. Rizwan Parvez, Shafiq R. Joty. 2023

95 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Wen-Ding Li
6. Ion Stoica
7. Swarat Chaudhuri
8. Rishabh Singh

In Information Systems

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Ion Stoica
6. Rishabh Singh
7. Mosharaf Chowdhury
8. Charles Sutton

In Code Generation & Program Synthesis

1. Armando Solar-Lezama
2. Koushik Sen
3. Rishabh Singh

Citations in the corpus

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “A few benchmarks specifically measure competitive programming, such as APPS (Hendrycks et al., 2021), CodeContests (Li et al., 2022), CodeScope (Yan et al., 2023), xCodeEval (Khan et al., 2023), and LeetCode-Hard (Shinn et al., 2023), and TACO (Li et al., 2023c).”
• A Survey on Large Language Models for Code Generation (2024, unknown): “Meanwhile, Chavan et al. [27] highlight the critical challenges of optimizing LLMs for faster and more memory-efficient, particularly in resource-constrained environments.”
• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, methodology): “To facilitate this evaluation metric, we developed a multilingual integrated execution testing environment, called Mul-tiCodeEngine , which can support 47 compiler/interpreter versions across the 14 programming languages involved in code generation, partially based on ExecEval (Khan et al., 2023).”
• Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval (2026, unknown): “5 Turbo [13].”
• Lost in Translation, Found in Summary: Summary-Driven Supervision for Cross-Language Code Retrieval (2026, unknown): “(2) To further assess SemSIR’s generalization ability in more challenging settings, we additionally adopt a large-scale test set derived from XCodeEval [17].”