Multi-task Learning based Pre-trained Language Model for Code Completion

By F. Liu, Ge Li, Yunfei Zhao, Zhi Jin. 2020

236 citations recorded by the source corpus.

Code completion is one of the most useful features in the Integrated Development Environments (IDEs), which can accelerate software development by suggesting the next probable token based on the contextual code in real-time. Recent studies have shown that statistical language modeling techniques can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from two major drawbacks: a) Existing research uses static embeddings, which map a word to the same vector regardless of its context. The differences in the meaning of a token in varying contexts are lost when each token is associated with a single representation; b) Existing language model based code completion models perform poor on completing identifiers, and the type information of the identifiers is ignored in most of these models. To address these challenges, in this paper, we develop a multi-task learning based pre-trained language model for code understanding and code generation with a Transformer-based neural architecture. We pre-train it with hybrid objective functions that incorporate both code understanding and code generation tasks. Then we fine-tune the pre-trained model on code completion. During the completion, our model does not directly predict the next token. Instead, we adopt multi-task learning to predict the token and its type jointly and utilize the predicted type to assist the token prediction. Experiments results on two real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.

Subfields: Computer Networks and Communications, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Software

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “…language processing and demonstrated remarkable abilities for tasks such as machine translation (Gulcehre et al., 2017; Baziotis et al., 2020), open-ended dialogue modeling (Hosseini-Asl et al., 2020; Thoppilan et al., 2022), and even code completion (Svyatkovskiy et al., 2019; Liu et al., 2020).”
Learning to Predict User-Defined Types (2023, unknown): “Multi-task learning is prevalent in machine learning ﬁelds of natural language processing [52], computer vision [53], and speech recognition [54], but is seldom used in software engineering [55], [56].”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “MTL has been applied successfully in both NLP [13] and source code analysis [32, 33].”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “programming languages (Kanade et al., 2020; Liu et al., 2020), Kanade et al.”
LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023, methodology): “Liu et al. (2020) and Guo et al. (2022) pre-train similar models CugLM and UniXcoder with multi-task learning by leveraging code structure for code completion.”
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (2023, unknown): “UniXCoder [18] and CugLM [24] incorporates multi-task learning strategies, and leverages code structures to enhance pretraining.”