CoTexT: Multi-task Learning with Code-Text Transformer

By Long Phan, H. Tran, Daniel Le, H. Nguyen, J. Anibal, Alec Peltekian, Yanfang Ye. 2021

160 citations recorded by the source corpus.

We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both “bimodal” and “unimodal” data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.

Subfields: Artificial Intelligence, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Multilingual training for Software Engineering (2021, methodology): “us for just Python [54], but is worse overall.”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, background): “Phan et al. (2021) add special symbols commonly seen in source code (such as [ , { and $ ) into the SentencePiece tokenizer to better encode programming languages.”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Further advancements are evident in CodeT5+ [286], while AlphaCode [156] and CoTexT [222] showcase the architecture’s adaptability to various SE tasks.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[34], Programming [35]–[39], Dialogue [40], Networking”
PanGu-Coder: Program Synthesis with Function-Level Language Modeling (2022, methodology): “…e.g. to the Biomedical [37, 46, 9], Legal [14], Cyber Security [2], and Finance [6] domains, while simultaneously expanding to include signals from modalities other than natural language, e.g. Vision [65, 18, 15, 13, 64, 25], Proteins [11], Time Series [71, 72, 55] and Code [42, 70, 26, 31, 54].”