Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study
By Saki Imai. 2022
207 citations recorded by the source corpus.
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.
Subfields: Experimental and Cognitive Psychology, Information Systems and Management, Code Generation & Program Synthesis, Human-Computer Interaction, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Experimental and Cognitive Psychology
- Brendan Dolan-Gavitt
- Siddharth Garg
- Ramesh Karri
- Gustavo Sandoval
- Teo Nys
- Hammond A. Pearce
- Michel C. Desmarais
In Information Systems and Management
In Code Generation & Program Synthesis
- Nasser Giacaman
- Viraj Kumar
- Brendan Dolan-Gavitt
- Paul Denny
- Siddharth Garg
- Ramesh Karri
- Gustavo Sandoval
- Teo Nys
In Human-Computer Interaction
- Brendan Dolan-Gavitt
- Paul Denny
- Siddharth Garg
- Ramesh Karri
- Gustavo Sandoval
- Teo Nys
- Hammond A. Pearce
In Software Engineering
Citations in the corpus
- Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, methodology): “Imai [21] tasked a group of developers (N=21) to implement code for a ‘minesweeper’ game.”
- Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (2022, background): “Outside of computing education, several recent studies have explored the potential impact of Copilot for developers [8, 15, 19].”
- GitHub Copilot AI pair programmer: Asset or Liability? (2022, result): “Similar to other researchers [38, 29, 51, 8], we can only investigate Copilot’s functionality in suggesting code for the provided prompt.”