PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers

By Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan. 2020

168 citations recorded by the source corpus.

Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PyMT5, the Python method text-to-text transfer transformer, which is trained to translate between all pairs of Python method feature combinations: a single model that can both predict whole methods from natural language documentation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling effort of a large-scale parallel corpus of 26 million Python methods and 7.7 million method-docstring pairs, demonstrating that for docstring and method generation, PyMT5 outperforms similarly-sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CodeSearchNet test set, our best model predicts 92.1% syntactically correct method bodies, achieved a BLEU score of 8.59 for method generation and 16.3 for docstring generation (summarization), and achieved a ROUGE-L F-score of 24.8 for method generation and 36.7 for docstring generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Competition-level code generation with AlphaCode (2022, background): “, 2020) have created a surge of interest in using transformer models for code retrieval, translation and generation (Chen et al., 2021; Clement et al., 2020; Feng et al., 2020), making significant progress on program synthesis challenges.”
Program Synthesis with Large Language Models (2021, methodology): “…models for code have shown to be effective for transfer learning across software engineering tasks, including CuBERT [Kanade et al., 2020], CodeBERT [Feng et al., 2020], PyMT5 [Clement et al., 2020], code2vec [Alon et al., 2019], and other T5 models trained on code [Mastropaolo et al., 2021].”
Evaluating Large Language Models Trained on Code (2021, methodology): “Popular language modeling objectives like masked language modeling (Devlin et al., 2018) and span prediction (Raffel et al., 2020) have also been adapted to train their programming counterparts CodeBERT (Feng et al., 2020) and PyMT5 (Clement et al., 2020).”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “, 2020), PyMT5 (Clement et al., 2020), and CodeT5 (Wang et al.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “[17] C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “PyMT5: Multi-mode translation of natural language and Python code with transformers,” in Proc.”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, background): “Some emerging work (Clement et al., 2020; Mas-tropaolo et al., 2021; Elnaggar et al., 2021) in the recent literature also explore the T5 framework on code, but they only focus on a limited subset of generation tasks and do not support understanding tasks like us.”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “While most of the prior approaches use task-specific representation learning, a few works (Alon et al., 2019b; Feng et al., 2020; Guo et al., 2021; Lachaux et al., 2020; Clement et al., 2020) attempted to learn transferable representations in an unsupervised fashion.”
PanGu-Coder: Program Synthesis with Function-Level Language Modeling (2022, methodology): “CodeT5 [70] and PYMT5 [22] were built on top of [58, T5] while UniXcoder [31] was based on UniLM [24].”
Jigsaw: Large Language Models meet Program Synthesis (2021, methodology): “for Python programming [15], general programming [11]) are rendered complementary to our efforts rather than competitive.”
GitHub Copilot AI pair programmer: Asset or Liability? (2022, unknown): “Novel Large Language Models (LLMs) with the transformer architecture recently achieved good performance in automatic program synthesis [6, 8, 9, 20].”
Measuring The Impact Of Programming Language Distribution (2023, background): “CodeT5 (Wang et al., 2021), PLBART (Ahmad et al., 2021), and additional works (Clement et al., 2020; Orlanski & Gittens, 2021; Chakraborty et al., 2022) examine training encoderdecoder models on code.”
An Analysis of the Automatic Bug Fixing Performance of ChatGPT (2023, methodology): “In recent years, several large-scale language models based on the Transformer architecture [7] have been introduced, such as CodeBERT [8], PyMT5 [9], and Codex [10], which can also process and extend source code and achieve comparable results to standard approaches on various coding tasks [11].”
CoDesc: A Large Code–Description Parallel Dataset (2021, methodology): “Along with the mentioned dataset, Clement et al. (2020) presented PyMT5, a text to text transformer that notably improved method generation and code summarization.”
METHODS2TEST: A dataset of focal methods mapped to test cases (2022, background): “Examples of these tasks include automating developers’ activities, such as: bug-fixing [5, 27], fault localization [28], writing comments [13] and documentation [6], code reviews [25], and merge conflict resolution [24].”
A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Programming Languages (2022, background): “2022); and several LLMs have been trained or fine-tuned on programming language text (Feng et al. 2020; Clement et al. 2020; Chen et al. 2021; Nijkamp et al. 2022; Fried et al. 2022; Xu et al. 2022).”
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (2022, methodology): “…a rising number of applications of pretrained language models (LMs) to program synthesis to exploit the contextual representations learned from massive data of codes and natural languages [Feng et al., 2020, Clement et al., 2020, Wang et al., 2021, Wang and Komatsuzaki, 2021, Chen et al., 2022].”