An Empirical Study of Code Smells in Transformer-based Code Generation Techniques

By Mohammed Latif Siddiq, Shafayat H. Majumder, M. R. Mim, Sourov Jajodia, Joanna C. S. Santos. 2022

102 citations recorded by the source corpus.

Prior works have developed transformer-based language learning models to automatically generate source code for a task without compilation errors. The datasets used to train these techniques include samples from open source projects which may not be free of security flaws, code smells, and violations of standard coding practices. Therefore, we investigate to what extent code smells are present in the datasets of coding generation techniques and verify whether they leak into the output of these techniques. To conduct this study, we used Pylint and Bandit to detect code smells and security smells in three widely used training sets (CodeXGlue, APPS, and Code Clippy). We observed that Pylint caught 264 code smell types, whereas Bandit located 44 security smell types in these three datasets used for training code generation techniques. By analyzing the output from ten different configurations of the open-source fine-tuned transformer-based GPT-Neo 125M parameters model, we observed that this model leaked the smells and non-standard practices to the generated source code. When analyzing GitHub Copilot's suggestions, a closed source code generation tool, we observed that it contained 18 types of code smells, including substandard coding patterns and 2 security smell types.

Subfields: Information Systems, Software, Model Architectures, Code Generation & Program Synthesis, Signal Processing, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Model Architectures

In Code Generation & Program Synthesis

In Signal Processing

In Software Engineering

Citations in the corpus

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, methodology): “This is contrary to literature [7, 8] which suggests that LLMs should be used with care due to their habit of suggesting vulnerable patterns.”