In-IDE Code Generation from Natural Language: Promise and Challenges

By Frank F. Xu, Bogdan Vasilescu, Graham Neubig. 2021

186 citations recorded by the source corpus.

A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code, especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries, but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this article, we perform the first comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking, “At the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?” To facilitate the study, we first develop a plugin for the PyCharm IDE that implements a hybrid of code generation and code retrieval functionality, and we orchestrate virtual environments to enable collection of many user events (e.g., web browsing, keystrokes, fine-grained code edits). We ask developers with various backgrounds to complete 7 varieties of 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Further analysis identifies several pain points that could improve the effectiveness of future machine learning-based code generation/retrieval developer assistants and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies on this topic, as well as development of better code generation models.

Subfields: Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, methodology): “Most past studies of the impacts of code generation models consider performance on a closed set of tasks in a simulated environment (Xu et al., 2021).”
A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, methodology): “Evaluating the capabilities of code synthesis and generation is not a novel problem and has been explored in both the Machine Learning (ML) [35] and Synthesis [20, 21, 30] communities.”
Grounded Copilot: How Programmers Interact with Code-Generating Models (2022, result): “Our study provides evidence for interaction modes like those in [Xu et al. 2021], where the natural language input is only temporary.”
Productivity assessment of neural code completion (2022, background): “Another study considered the benefits of using a plugin converting natural language prompts to code [20].”
Automatic Document Sketching: Generating Drafts from Analogous Texts (2021, background): “, entity type) or a retrieved sample content for a gap (Xu et al., 2021) could help make sketches more interpretable.”
The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development (2023, methodology): “[103, 105, 109]) conducted quantitative examinations of the use of LLMs in code work; our study is akin to Weisz et al.”
Synchromesh: Reliable code generation from pre-trained language models (2022, background): “Indeed, users of current natural language-to-code systems report that models often produce code that is unrelated to their query (Xu et al., 2021).”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, methodology): “Currently, there are only a few studies on how programmers use such imperfect code generation tools [44, 47].”