The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
By Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer. 2023
697 citations recorded by the source corpus.
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Subfields: Computer Science Applications, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Andrew Luxton-Reilly
- James Prather
- Brett A. Becker
- Daniel Rock
In Information Systems
- Paul Denny
- Juho Leinonen
- Arto Hellas
- Sami Sarsa
- Shin Yoo
- Mark Harman
- Andrew Luxton-Reilly
- James Prather
In Evaluation & Benchmarks
- Juho Leinonen
- Paul Denny
- Daniel Rock
- James Prather
- Brett A. Becker
- Mark Harman
- Sami Sarsa
- Arto Hellas
In Code Generation & Program Synthesis
In Software Engineering
Citations in the corpus
- GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “This could lower the cost and engineering expertise required to create such tools, potentially accelerating LLM adoption and integration even further (Chen et al., 2021; Peng et al., 2023).”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “Similarly, Peng et al. [39] reported that programmers could complete a non-trivial task (implementing an HTTP server in JavaScript) 56% faster when paired with GitHub Copilot, compared to the control group that did not receive any such support.”
- The shaky foundations of large language models and foundation models for electronic health records (2023, background): “Metrics include user satisfaction, engagement, system usability scale scores, qualitative interview feedback, and the time/effort required to achieve stated goals 96 – 98 .”
- AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025, neutral): “Moreover, applications are no longer confined to narrow domains like simulations or logistics, but now extend to broad practical settings demanding real-time reasoning and adaptive control.”
- Computing Education in the Era of Generative AI (2023, background): “…source code automatically from natural language descriptions of problems is likely to greatly improve the productivity of professional developers [28], and is being actively explored by well-funded entities such as OpenAI (ChatGPT, Codex, GPT-x ), Amazon (CodeWhisperer), and Google (AlphaCode,…”
- ReservoirChat: Interactive Documentation Enhanced with LLM and Knowledge Graph for ReservoirPy (2025, unknown): “A notable example is GitHub Copilot [8], an assistant designed to provide suggestions, error corrections, and autocompletions of code.”