CodeTrans: Towards Cracking the Language of Silicone's Code Through Self-Supervised Deep Learning and High Performance Computing

By Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas B. Fehér, Christoph Angerer, Silvia Severini, Florian Matthes, Burkhard Rost. 2021

85 citations recorded by the source corpus.

Currently, a growing number of mature natural language processing applications make people's life more convenient. Such applications are built by source code - the language in software engineering. However, the applications for understanding source code language to ease the software engineering process are under-researched. Simultaneously, the transformer model, especially its combination with transfer learning, has been proven to be a powerful technique for natural language processing tasks. These breakthroughs point out a promising direction for process source code and crack software engineering tasks. This paper describes CodeTrans - an encoder-decoder transformer model for tasks in the software engineering domain, that explores the effectiveness of encoder-decoder transformer models for six software engineering tasks, including thirteen sub-tasks. Moreover, we have investigated the effect of different training strategies, including single-task learning, transfer learning, multi-task learning, and multi-task learning with fine-tuning. CodeTrans outperforms the state-of-the-art models on all the tasks. To expedite future works in the software engineering domain, we have published our pre-trained models of CodeTrans. https://github.com/agemagician/CodeTrans

Subfields: Information Systems, Software, Representation & Self-Supervised Learning, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Chun Xia
2. Lingming Zhang
3. Steven C. H. Hoi
4. Weishi Wang
5. Shafiq R. Joty
6. Yue Wang
7. Yinlin Deng
8. Soren Dunn

In Software

1. Chun Xia
2. Lingming Zhang
3. Steven C. H. Hoi
4. Yue Wang
5. Yinlin Deng
6. Soren Dunn

In Representation & Self-Supervised Learning

1. Steven C. H. Hoi
2. Shafiq R. Joty
3. Lingming Zhang
4. Junnan Li

In Code Generation & Program Synthesis

1. Chun Xia
2. Steven C. H. Hoi
3. Lingming Zhang
4. Yue Wang

Citations in the corpus

• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, background): “Some emerging work (Clement et al., 2020; Mas-tropaolo et al., 2021; Elnaggar et al., 2021) in the recent literature also explore the T5 framework on code, but they only focus on a limited subset of generation tasks and do not support understanding tasks like us.”
• Less training, more repairing please: revisiting automated program repair via zero-shot learning (2022, methodology): “CodeTrans [22], CodeT5 [72] and PLBART [4] are unified encoder-decoder models [63] which uses denoising sequence-to-sequence training objectives to pre-train both the encoder and decoder for various coding tasks.”