An Empirical Study of the Non-Determinism of ChatGPT in Code Generation

By Shuyin Ouyang, Jie M. Zhang, Mark Harman, Meng Wang. ACM Transactions on Software Engineering and Methodology, 2023

305 citations recorded by the source corpus.

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/dc7de606ecf7c65abc26396629b633e18d8704d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “LLMs are inherently nondeterministic: the same prompt produces different answers on different inference executions (unless the temperature is set to zero, which has often been found to be suboptimal over multiple executions) [9].”
The Metacognitive Demands and Opportunities of Generative AI (2023, background): “Error analysis [31] Mistakes made during task performance are analyzed to understand metacognitive flexibility.”
Mathematical Capabilities of ChatGPT (2023, background): “From the viewpoint of reproducing our dataset, this is also not an issue, since even for identical prompts, (Chat)GPT itself is non-deterministic; this continues even if its temperature parameter in the API is set to zero [50, 51].”