Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models

By Priyan Vaithilingam, Tianyi Zhang, Elena L. Glassman. 2022

744 citations recorded by the source corpus.

Recent advances in Large Language Models (LLM) have made automatic code generation possible for real-world programming tasks in general-purpose programming languages such as Python. However, there are few human studies on the usability of these tools and how they fit the programming workflow. In this work, we conducted a within-subjects user study with 24 participants to understand how programmers use and perceive Copilot, a LLM-based code generation tool. We found that, while Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks, since Copilot often provided a useful starting point and saved the effort of searching online. However, participants did face difficulties in understanding, editing, and debugging code snippets generated by Copilot, which significantly hindered their task-solving effectiveness. Finally, we highlighted several promising directions for improving the design of Copilot based on our observations and participants’ feedback.

Subfields: Computer Networks and Communications, Information Systems, Large Language Models, Code Generation & Program Synthesis, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

  1. Brendan Dolan-Gavitt
  2. Shin Yoo
  3. Mark Harman
  4. Siddharth Garg
  5. Silvio Savarese
  6. Foutse Khomh
  7. Ramesh Karri
  8. Eirini Kalliamvakou

In Information Systems

  1. Paul Denny
  2. Juho Leinonen
  3. Arto Hellas
  4. Sami Sarsa
  5. Shin Yoo
  6. Mark Harman
  7. Siddharth Garg
  8. Ramesh Karri

In Large Language Models

  1. Paul Denny
  2. Juho Leinonen
  3. Sami Sarsa
  4. Arto Hellas
  5. Silvio Savarese
  6. Brendan Dolan-Gavitt
  7. James Prather
  8. Brett A. Becker

In Code Generation & Program Synthesis

  1. Paul Denny
  2. Juho Leinonen
  3. Sami Sarsa
  4. Arto Hellas
  5. Brendan Dolan-Gavitt
  6. Mark Harman
  7. Siddharth Garg
  8. Silvio Savarese

In Human-Computer Interaction

  1. Paul Denny
  2. Sami Sarsa
  3. Juho Leinonen
  4. Arto Hellas
  5. Brendan Dolan-Gavitt
  6. Siddharth Garg
  7. Silvio Savarese
  8. Ramesh Karri

Citations in the corpus