When Code Completion Fails: A Case Study on Real-World Completions

By Vincent J. Hellendoorn, Sebastian Proksch, Harald C. Gall, Alberto Bacchelli. 2019

86 citations recorded by the source corpus.

Code completion is commonly used by software developers and is integrated into all major IDE's. Good completion tools can not only save time and effort but may also help avoid incorrect API usage. Many proposed completion tools have shown promising results on synthetic benchmarks, but these benchmarks make no claims about the realism of the completions they test. This lack of grounding in real-world data could hinder our scientific understanding of developer needs and of the efficacy of completion models. This paper presents a case study on 15,000 code completions that were applied by 66 real developers, which we study and contrast with artificial completions to inform future research and tools in this area. We find that synthetic benchmarks misrepresent many aspects of real-world completions; tested completion tools were far less accurate on real-world data. Worse, on the few completions that consumed most of the developers' time, prediction accuracy was less than 20% -- an effect that is invisible in synthetic benchmarks. Our findings have ramifications for future benchmarks, tool design and real-world efficacy: Benchmarks must account for completions that developers use most, such as intra-project APIs; models should be designed to be amenable to intra-project data; and real-world developer trials are essential to quantifying performance on the least predictable completions, which are both most time-consuming and far more typical than artificial data suggests. We publicly release our preprint [https://doi.org/10.5281/zenodo.2565673] and replication data and materials [https://doi.org/10.5281/zenodo.2562249].

Subfields: Computer Networks and Communications, Information Systems, Software, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

In Code Generation & Program Synthesis

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “, 2020), as well as from synthetic data to real data (including sim2real) (Wang et al., 2015; Ganin & Lempitsky, 2015; Richter et al., 2016; Peng et al., 2018; Hellendoorn et al., 2019; Xu et al., 2020; Bellemare et al., 2020), as synthetic data can be easier to obtain than real data.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “While previous work [42] found that closed NLMs underperformed on identifiers, we find that our BPE NLMs do not.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Another critical challenge is the subpopulation shift from training time to real usage: for instance, Hellendoorn et al. (2019) show that existing code completion systems, typically trained as language models on source code, perform poorly on real completion instances that developers use the most…”