Productivity assessment of neural code completion

By Albert Ziegler, Eirini Kalliamvakou, Shawn Simister, Ganesh Sittampalam, Alice Li, Andrew Rice, Devon Rifkin, Edward Aftandilian. 2022

303 citations recorded by the source corpus.

Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers’ productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.

Subfields: Information Systems, Information Systems and Management, Materials Chemistry, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Systems and Management

In Materials Chemistry

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “New products such as GitHub Copilot 1 , Amazon CodeWhisperer 2 , and Tabnine 3 built on code generation models are growing in popularity with developers [7].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, background): “Research studies [46] also show that 88% of users of GitHub Copilot—a paid service powered by Codex—feel more productive when coding with it.”
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “Functionality (RQ1): our results corroborate recent studies that have suggested that LLM assistants improve developer productivity [22, 23].”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, background): “New products such as GitHub Copilot1, Amazon CodeWhisperer2, and Tabnine3 built on code generation models are growing in popularity with developers [8].”
GitHub Copilot AI pair programmer: Asset or Liability? (2022, background): “Authors in [37] surveyed 2631 developers about the impact of Copilot on their productivity”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, background): “…et al., 2022), program repair (Berabi et al., 2021), code translation (Roziere et al., 2020, 2021b), and code retrieval (Wan et al., 2019; Parvez et al., 2021) are having a profound impact on increasing developer productivity (Ziegler et al., 2022) and aiding educators (Finnie-Ansley et al., 2022).”
Improving Code Generation by Training with Natural Language Feedback (2023, background): “Effective program synthesis can not only improve the efﬁciency of software developers (Ziegler et al., 2022), but also increase the accessibility of writing code in general.”
Augmenting large language models with chemistry tools (2023, background): “These specialized tools provide exact answers, thereby compensating for the inherent deficiencies of LLMs in specific domains and enhancing their overall performance and applicability.”