Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?

By Jean-Baptiste Döderlein, Nguessan Hermann Kouadio, M. Acher, D. Khelladi, Benoît Combemale. 2022

37 citations recorded by the source corpus.

Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently gained attention in code assistants, which generate programs from a natural language task description (prompt). They have the potential to save time and effort but remain poorly understood, limiting their optimal use. In this article, we investigate the impact of input variations on two configurations of a language model, focusing on parameters such as task description, surrounding context, model creativity, and the number of generated solutions. We design specific operators to modify these inputs and apply them to three LLM-based code assistants (Copilot, Codex, StarCoder2) and two benchmarks representing algorithmic problems (HumanEval, LeetCode). Our study examines whether these variations significantly affect program quality and how these effects generalize across models. Our results show that varying input parameters can greatly improve performance, achieving up to 79.27% success in one-shot generation compared to 22.44% for Codex and 31.1% for Copilot in default settings. Actioning this potential in practice is challenging due to the complex interplay in our study - the optimal settings for temperature, prompt, and number of generated solutions vary by problem. Reproducing our study with StarCoder2 confirms these findings, indicating they are not model-specific. We also uncover surprising behaviors (e.g., fully removing the prompt can be effective), revealing model brittleness and areas for improvement.

Subfields: Artificial Intelligence, Hardware and Architecture, Large Language Models, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Large Language Models

In Evaluation & Benchmarks

Mark Harman

In Code Generation & Program Synthesis

Citations in the corpus

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “6 based on the results reported in the original Codex work [3] as well as in [47], which show that the best pass@10 rate is at temperature=0.”
Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “However, as already found [57], the results are highly problem-specific, so a one-size-fits-all approach is unrealistic.”