An Empirical Evaluation of GitHub Copilot's Code Suggestions

By Nhan Nguyen, Sarah Nadi. IEEE Working Conference on Mining Software Repositories, 2022

395 citations recorded by the source corpus.

GitHub and OpenAI recently launched Copilot, an “AI pair programmer” that utilizes the power of Natural Language Processing, Static Analysis, Code Synthesis, and Artificial Intelligence. Given a natural language description of the target functionality, Copilot can generate corresponding code in several programming languages. In this paper, we perform an empirical study to evaluate the correctness and understandability of Copilot's suggested code. We use 33 LeetCode questions to create queries for Copilot in four different programming languages. We evaluate the correctness of the corresponding 132 Copilot solutions by running LeetCode's provided tests, and evaluate understandability using SonarQube's cyclomatic complexity and cognitive complexity metrics. We find that Copilot's Java suggestions have the highest correctness score (57%) while JavaScript is the lowest (27%). Overall, Copilot's suggestions have low complexity with no notable differences between the programming languages. We also find some potential Copilot shortcomings, such as generating code that can be further simplified and code that relies on undefined helper methods.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/cdfe9580f63070f311151444f9df32818cc858bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “Evaluations have often relied on small programming problems from programming courses [86], synthetically generated problem sets [87], and online judging platforms such as Leetcode [28], [64], [88].”
The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development (2023, background): “this model have shown that, although the quality of its outputs is quite good, those outputs may still be problematic [57].”
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (2022, background): “Outside of computing education, several recent studies have explored the potential impact of Copilot for developers [8, 13, 17].”
GitHub Copilot AI pair programmer: Asset or Liability? (2022, methodology): “Nguyen and Nadi [38] compared the complexity of Copilot’s solutions in diﬀerent programming languages for several LeetCode questions, besides their correctness.”
Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, background): “…complex, emergent behaviors, such as abilities to write code in programming languages, generate stories, poems, essays, and other texts, and demonstrate strong performance in certain reasoning tasks (Chen et al., 2021; Nguyen & Nadi, 2022; Bubeck et al., 2023; Katz et al., 2023; Wei et al., 2022a).”