ReCode: Robustness Evaluation of Code Generation Models

By Shiqi Wang, Zheng Li, Haifeng Qian, Cheng Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia and 4 more. 2022

161 citations recorded by the source corpus.

Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model’s robustness performance. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

In Signal Processing

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “Their variants have been proposed to cover more languages, (Wang et al., 2022a; Zheng et al., 2023; Cassano et al., 2022; Athiwaratkun et al., 2022).”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “In addition, we perturb the natural language descriptions in docstrings with the back-translation method of NL-Augmenter [17], inspired by ReCode [57].”
Measuring The Impact Of Programming Language Distribution (2023, background): “Related Works Code Evaluation Existing code benchmarks have primarily focused on surface matching evaluation (Lu et al., 2021; Yin et al., 2018; Wang et al., 2022b; Husain et al., 2019).”
A Survey on Large Language Models for Code Generation (2024, unknown): “These real-world applications illustrate how LLMs are revolutionizing industries and unlocking innovative solutions in diverse domains.”