Grounded Copilot: How Programmers Interact with Code-Generating Models

By Shraddha Barke, Michael B. James, Nadia Polikarpova. Proc. ACM Program. Lang., 2022

620 citations recorded by the source corpus.

Powered by recent advances in code-generating models, AI assistants like Github Copilot promise to change the face of programming forever. But what is this new face of programming? We present the first grounded theory analysis of how programmers interact with Copilot, based on observing 20 participants—with a range of prior experience using the assistant—as they solve diverse programming tasks across four languages. Our main finding is that interactions with programming assistants are bimodal: in acceleration mode, the programmer knows what to do next and uses Copilot to get there faster; in exploration mode, the programmer is unsure how to proceed and uses Copilot to explore their options. Based on our theory, we provide recommendations for improving the usability of future AI programming assistants.

Subfields: Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/fb9a88ac3c4795e3fcf5de926516c59463b42e67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

Citations in the corpus

A Survey on Large Language Models for Code Generation (2024, neutral): “The tasks are further evaluated by other LLMs and cross-checked by 7 experts, with the resulting average human performance at 97%.”
Multi-lingual Evaluation of Code Generation Models (2022, background): “Code completion by machine-learning models has great potential to improve developer productivity (Barke et al., 2022).”
The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development (2023, background): “Developers would communicate with Copilot by typing comments and seeing what Copilot generated in response.”
Computing Education in the Era of Generative AI (2023, background): “Other recent work has shown that developers are more successful working with Copilot when they decompose larger programming statements into smaller tasks and then explicitly prompt Copilot for each of the subtasks [1, 18].”
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book (2022, methodology): “currently power tools like Github Copilot, Amazon Codewhisperer, and Tabnine which can generate code based on natural language specifications provided by a software developer [3, 6, 37].”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023, background): “Some focused on finding interaction patterns [32, 148, 176] while others focused more on how productivity is impacted by the use of models [101, 131], whether code produced when using AI code generators is less secure than when not [162], and how students use code explanations generated by”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “…related problems in the same notebook context to better reﬂect the real-world scenario where data scientists prompt LLMs using ephemeral NL comments for assistance with complex data wrangling or EDA tasks (Barke et al., 2022), therefore making it ideal to evaluate LLMs in this realistic setting.”
CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context (2022, positive): “In recent years, language models for source code like Codex (Chen et al., 2021) and CodeGen (Ni-jkamp et al., 2023) have shown promising performance in code completion tasks and have great potential to improve developer productivity (Barke et al., 2023).”