Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
By Priyan Vaithilingam, Tianyi Zhang, Elena L. Glassman. 2022
744 citations recorded by the source corpus.
Recent advances in Large Language Models (LLM) have made automatic code generation possible for real-world programming tasks in general-purpose programming languages such as Python. However, there are few human studies on the usability of these tools and how they fit the programming workflow. In this work, we conducted a within-subjects user study with 24 participants to understand how programmers use and perceive Copilot, a LLM-based code generation tool. We found that, while Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks, since Copilot often provided a useful starting point and saved the effort of searching online. However, participants did face difficulties in understanding, editing, and debugging code snippets generated by Copilot, which significantly hindered their task-solving effectiveness. Finally, we highlighted several promising directions for improving the design of Copilot based on our observations and participants’ feedback.
Subfields: Computer Networks and Communications, Information Systems, Large Language Models, Code Generation & Program Synthesis, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Brendan Dolan-Gavitt
- Shin Yoo
- Mark Harman
- Siddharth Garg
- Silvio Savarese
- Foutse Khomh
- Ramesh Karri
- Eirini Kalliamvakou
In Information Systems
In Large Language Models
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- Silvio Savarese
- Brendan Dolan-Gavitt
- James Prather
- Brett A. Becker
In Code Generation & Program Synthesis
- Paul Denny
- Juho Leinonen
- Sami Sarsa
- Arto Hellas
- Brendan Dolan-Gavitt
- Mark Harman
- Siddharth Garg
- Silvio Savarese
In Human-Computer Interaction
- Paul Denny
- Sami Sarsa
- Juho Leinonen
- Arto Hellas
- Brendan Dolan-Gavitt
- Siddharth Garg
- Silvio Savarese
- Ramesh Karri
Citations in the corpus
- Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, methodology): “[20] measured productivity from a group of developers (N=24) completing code tasks in Python.”
- Productivity assessment of neural code completion (2022, methodology): “A previous study comparing GitHub Copilot against IntelliCode with 25 participants found no significant correlation between task completion times and survey responses [14].”
- The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023, background): “Further, novice learners of programming may become overwhelmed and confused by generated code, which could add to the high levels of frustration that are common in introductory programming courses [70, 176].”
- Computing Education in the Era of Generative AI (2023, background): “Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “This approach leverages the natural language processing capabilities and contextual understanding of LLMs to identify malicious software.”
- GitHub Copilot AI pair programmer: Asset or Liability? (2022, methodology): “There are studies on Copilot that used easy but more practical programming tasks than course assignments, such as tasks in [51] (i.e., editing a CSV file), but these types of tasks need less problem-solving effort compared to the assignments of a programming course in our selected dataset.”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “For example, Vaithilingam et al. [200] reported on the difficulties 24 participants had in understanding, editing, and debugging the Copilot-generated code, while Feldt et al. [133] proposed a hierarchy of design architecture for LLM-based software testing agents.”