A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation

By Antonio Valerio Miceli Barone, Rico Sennrich. 2017

179 citations recorded by the source corpus.

Automated documentation of programming source code and automated code generation from natural language are challenging tasks of both practical and scientific interest. Progress in these areas has been limited by the low availability of parallel corpora of code and natural language descriptions, which tend to be small and constrained to specific domains. In this work we introduce a large and diverse parallel corpus of a hundred thousands Python functions with their documentation strings (“docstrings”) generated by scraping open source repositories on GitHub. We describe baseline results for the code documentation and code generation tasks obtained by neural machine translation. We also experiment with data augmentation techniques to further increase the amount of training data. We release our datasets and processing scripts in order to stimulate research in these areas.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “Barone & Sennrich (2017) proposed a large training and evaluation dataset consisting of Python declarations, docstrings, and bodies scraped from GitHub.”
The adverse effects of code duplication in machine learning models of code (2018, methodology): “Here, the most recent approach is that of Barone and Sennrich [7] that use neural machine translation to “translate” code to documentation.”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “Research on Source Code Generating Models PD (c|C (c)) (Sorted Alphabetically), Describing the Process of Generating Source Code Reference Type Representation PD Application Aggarwal et al. [2] PD (c |s) Token Phrase Migration Allamanis and Sutton [11] PD (c) Token n-gram — Allamanis et al. [5] PD (”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, neutral): “…like type prediction (Mir et al., 2022; Wei et al., 2023a; Malik et al., 2019), code summarization (LeClair et al., 2019; Iyer et al., 2016; Barone and Sennrich, 2017; Hasan et al., 2021; Alon et al., 2018), code security (Liguori et al., 2022; Pearce et al., 2022; Tony et al., 2023), etc.…”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, methodology): “The n-gram accuracy (e.g. 4-gram BLEU) is the most popular evaluation method for code synthesis (Karaivanov, Raychev, and Vechev 2014; Barone and Sennrich 2017), based on the token overlapping between the hypothesis and the reference.”
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, methodology): “…to extract features like the Ab-stract Syntax Tree for modeling (treating code and natural language as separate modalities), we follow examples like Barone and Sennrich (2017) and treat P YTHON and its docstrings as fundamentally no different than other ‘natural’ languages, representing both…”
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, background): “, [15, 44, 46, 48]), while there has been research on mining from other data sources such as API documentation [5, 6, 24], code comments [43], specialized sites [32], parameter/method/class names [1, 37], and developer mailing lists [31].”
Neural Program Synthesis with Priority Queue Training (2018, neutral): “…al., 2016; Gaunt et al., 2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016; Schaechtle et al., 2016; Zaremba et al., 2016; Barone & Sennrich, 2017; Beltramelli, 2017; Cai et al., 2017; Devlin et al., 2017; Hu et al., 2017; Guu et al., 2017; Johnson et al.; Liang et al.,…”
TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search (2020, methodology): “To evaluate the eectiveness and eciency of TranS3, we conduct a set of experiments based on the GitHub dataset in [29] which includes over 120,000 code snippets of Python functions and their corresponding comments.”
Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, methodology): “We evaluate the performance of our proposed method using the dataset in [6], which is obtained from a popular open source projects hosting platform, GitHub1.”
Structured Neural Summarization (2018, methodology): “However, 2017), we measure the BLEU score for all models.”
StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow (2018, methodology): “In order to train more advanced yet data-hungry models, researchers have collected data either automatically from online communities [2, 5, 21, 22, 26, 45, 57, 64] or with human intervention [16, 35].”
Typilus: neural type hints (2020, unknown): “com/allenai/allennlp/pull/3376 versa [8, 11, 24, 35].”
CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021, background): “Husain et al. (2019), Gu et al. (2018) and Miceli Barone and Sennrich (2017) collect large-scale unlabelled text-code pairs by leveraging human-leaved comments in code functions from GitHub.”
Pre-trained Contextual Embedding of Source Code (2019, methodology): “This provides parallel corpora between code and natural language sentences that have been used for machine translation between the two (Barone & Sennrich, 2017), detecting uninformative docstrings (Louis et al., 2018) and to evaluate their utility to provide supervision in neural code search…”
Retrieval-based Neural Source Code Summarization (2020, background): “[6], which contains Python functions and their comments from open source repositories in GitHub2.”
Code Generation as a Dual Task of Code Summarization (2019, methodology): “The original Python dataset is collected by Barone and Sennrich [2017], consisting of about 110K parallel samples and about 160K code-only samples.”