A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages
By Federico Cassano, John Gouwar, Daniel Nguyen, S. Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman and 3 more. 2022
31 citations recorded by the source corpus.
Subfields: Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Shuyan Zhou
- Graham Neubig
- Qian Liu
- Harm de Vries
- Manan Dey
- Niklas Muennighoff
- Danish Contractor
- Terry Yue Zhuo
In Code Generation & Program Synthesis
- Shuyan Zhou
- Qian Liu
- Graham Neubig
- Raymond Li
- Carolyn Jane Anderson
- Arjun Guha
- Dzmitry Bahdanau
- Uri Alon
Citations in the corpus
- SantaCoder: don't reach for the stars! (2023, positive): “We use the MultiPL-E text2code benchmark Cassano et al. (2022), which is derived from HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) (the “sanitized” subset of MBPP.).”
- The Stack: 3 TB of permissively licensed source code (2022, background): “Recently, these benchmarks have been extended from Python to other programming languages (Athiwaratkun et al., 2022; Cassano et al., 2022).”
- CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023, methodology): “(c) We pretrain and release five language-specific CodeBERT models to use with our publicly available code, for Java, Python, C, C++, and JavaScript.”
- A Survey on Large Language Models for Code Generation (2024, unknown): “All of these, except the 70B model, have been trained on 500B tokens of code and code-related data, whereas the 70B model required 1 trillion tokens as they continue to push the limits on current state-of-the-art computing resources.”
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, unknown): “…namely HumanEval (Chen et al., 2021b), HumanEval+ (Liu et al., 2023), MBPP (Austin et al., 2021), DS-100 (Lai et al., 2022), and MultiPL-E (Cassano et al., 2022), demonstrate that our WizardCoder models outperform all other open-source Code LLMs (before August 24, 2023), achieving…”