Learning Autocompletion from Real-World Datasets

By Gareth Ari Aye, Seohyun Kim, Hongyu Li. 2020

37 citations recorded by the source corpus.

Code completion is a popular software development tool integrated into all major IDEs. Many neural language models have achieved promising results in completion suggestion prediction on synthetic benchmarks. However, a recent study When Code Completion Fails: a Case Study on Real-World Completions demonstrates that these results may not translate to improvements in real-world performance. To combat this effect, we train models on real-world code completion examples and find that these models outperform models trained on committed source code and working version snapshots by 12.8% and 13.8% accuracy respectively. We observe this improvement across modeling technologies and show through A/B testing that it corresponds to a 6.2% increase in programmers' actual autocompletion usage. Furthermore, our study characterizes a large corpus of logged autocompletion usages to investigate why training on real-world examples leads to stronger models.

Subfields: Computer Networks and Communications, Information Systems, Software, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Sven Gowal
2. Eirini Kalliamvakou
3. Maliheh Izadi
4. Roberta Gismondi
5. Georgios Gousios
6. Joshua Achiam
7. Xinyun Chen
8. Pushmeet Kohli

In Information Systems

1. Xinyun Chen
2. Eirini Kalliamvakou
3. Oriol Vinyals
4. Hung Le
5. Maliheh Izadi
6. Roberta Gismondi
7. Georgios Gousios
8. Pushmeet Kohli

In Software

1. Xinyun Chen
2. Pushmeet Kohli
3. Akhilesh Deepak Gotmare
4. Maliheh Izadi
5. Nate Kushman
6. Ilya Sutskever
7. Silvio Savarese
8. Steven C. H. Hoi

In Natural Language Processing

1. Mohit Bansal
2. Quoc V. Le
3. Yoshua Bengio
4. Kyunghyun Cho
5. Jianfeng Gao
6. Graham Neubig
7. Christopher D. Manning
8. Oriol Vinyals

In Code Generation & Program Synthesis

1. Xinyun Chen
2. Pushmeet Kohli
3. Akhilesh Deepak Gotmare
4. Maliheh Izadi
5. Ilya Sutskever
6. Silvio Savarese
7. Steven C. H. Hoi
8. Hung Le

Citations in the corpus

• Competition-level code generation with AlphaCode (2022, background): “More recent intelligent code completion systems can learn from history (Robbes and Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
• Evaluating Large Language Models Trained on Code (2021, background): “Aye et al. (2021) built an internal auto-complete tool for Face-book, and found that training on accepted user completions boosted system performance.”
• CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, background): “[7], trained models on realworld code completion examples of an internal dataset (Facebook).”
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (2022, methodology): “In the same domain, Bahdanau et al. [2016] introduced an actor-critic framework [Sutton, 1984, Konda and Tsitsiklis, 1999]. In visual captioning domains, Rennie et al. [2017], Wang et al. [2018] proposed to use RL to optimize image captioning models using variants of CIDEr scores [Vedantam et al., 2”
• Productivity assessment of neural code completion (2022, background): “An alternative measure is that of DCPU (Daily Completions accepted Per User) for which a value of around 20 has been reported [3, 21].”