Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study

By Saki Imai. 2022

207 citations recorded by the source corpus.

This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.

Subfields: Experimental and Cognitive Psychology, Information Systems and Management, Code Generation & Program Synthesis, Human-Computer Interaction, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

  1. Brendan Dolan-Gavitt
  2. Siddharth Garg
  3. Ramesh Karri
  4. Gustavo Sandoval
  5. Teo Nys
  6. Hammond A. Pearce
  7. Michel C. Desmarais

In Information Systems and Management

  1. Brendan Dolan-Gavitt
  2. Siddharth Garg
  3. Ramesh Karri
  4. Gustavo Sandoval
  5. Teo Nys
  6. Hammond A. Pearce

In Code Generation & Program Synthesis

  1. Nasser Giacaman
  2. Viraj Kumar
  3. Brendan Dolan-Gavitt
  4. Paul Denny
  5. Siddharth Garg
  6. Ramesh Karri
  7. Gustavo Sandoval
  8. Teo Nys

In Human-Computer Interaction

  1. Brendan Dolan-Gavitt
  2. Paul Denny
  3. Siddharth Garg
  4. Ramesh Karri
  5. Gustavo Sandoval
  6. Teo Nys
  7. Hammond A. Pearce

In Software Engineering

  1. Brendan Dolan-Gavitt
  2. Siddharth Garg
  3. Ramesh Karri
  4. Gustavo Sandoval
  5. Teo Nys
  6. Hammond A. Pearce

Citations in the corpus