Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks

By Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader-Palacio, Denys Poshyvanyk, Rocco Oliveto, Gabriele Bavota. 2021

285 citations recorded by the source corpus.

Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task (e.g., filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task (e.g., language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.

Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Program Synthesis with Large Language Models (2021, methodology): “…models for code have shown to be effective for transfer learning across software engineering tasks, including CuBERT [Kanade et al., 2020], CodeBERT [Feng et al., 2020], PyMT5 [Clement et al., 2020], code2vec [Alon et al., 2019], and other T5 models trained on code [Mastropaolo et al., 2021].”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, unknown): “Mastropaolo et al. [45], [131] pre-trained a T5 model on a dataset composed of natural language English text and source code.”
Multilingual training for Software Engineering (2021, methodology): “Deep learning models have been widely applied to code summarization, with papers reporting substantial gains in performance over recent years [1, 2, 7, 18, 19, 24–28, 30, 38–42, 50, 54, 55, 66, 67, 69–74].”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “We are aware of several recent research efforts in which LLMs are used for test generation [23], [24], [25], [26], [27], [28], [29].”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, methodology): “Though these methods have shown promising results, they still suffer from low accuracy and are less reliable [9, 29, 40].”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “model that has been used to support several code-related tasks [43], including code completion [12].”
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, background): “[19], [20] leverages advanced deep learning (DL) techniques, especially large language models (LLMs), to generate unit tests.”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, methodology): “Within the field of SE, this ability has been successfully applied to tasks like code summarization [5, 79, 186].”