Mapping Language to Code in Programmatic Context

By Srini Iyer, Ioannis Konstas, Alvin Cheung, Luke S. Zettlemoyer. 2018

181 citations recorded by the source corpus.

Source code is rarely written in isolation. It depends significantly on the programmatic context, such as the class that the code would reside in. To study this phenomenon, we introduce the task of generating class member functions given English documentation and the programmatic context provided by the rest of the class. This task is challenging because the desired code can vary greatly depending on the functionality the class provides (e.g., a sort function may or may not be available when we are asked to “return the smallest element” in a particular member variable list). We introduce CONCODE, a new large dataset with over 100,000 examples consisting of Java classes from online code repositories, and develop a new encoder-decoder architecture that models the interaction between the method documentation and the class environment. We also present a detailed error analysis suggesting that there is significant room for future work on this task.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Program Synthesis with Large Language Models (2021, neutral): “, 2016], and generating function text from docstrings [Iyer et al., 2018].”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, neutral): “In contrast, a few previous works instantiate benchmarks that measure the ability to generate programs given surrounding program context beyond the target program, such as variables and other methods (Iyer et al., 2018) or alternating “cells” of preceding code and text blocks (Agashe et al.”
SantaCoder: don't reach for the stars! (2023, neutral): “Many single-language benchmarks for evaluating code completion exist (Kulal et al., 2019; Iyer et al., 2018; Zhong et al., 2017; Yu et al., 2018; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Athiwaratkun et al., 2022; Lai et al., 2022).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, neutral): “The CONCODE (Iyer et al., 2018) dataset tested generation of Java class methods.”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, neutral): “We employ the Concode data (Iyer et al., 2018) in Java where the input contains both NL texts and class environment contexts, and the output is a function.”
Unified Pre-training for Program Understanding and Generation (2021, positive): “We fine-tune PLBART on the Concode dataset (Iyer et al., 2018), where the input is a text describing class member functions in Java and class environment, the output is the target function.”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, positive): “CodeXGLUE includes eight previously proposed datasets — BigCloneBench [65], POJ-104 [46], Devign [91], PY150 [57], Github Java Corpus [1], Bugs2Fix [69], CONCODE [34], and CodeSearchNet [32]— but also newly introduced datasets that are marked in Table 1.”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, neutral): “[14] generate code from natural language within the context of existing methods, whereas Allamanis et al.”
Measuring Coding Challenge Competence With APPS (2021, neutral): “We compared to several datasets in the (Kulal et al., 2019; Yu et al., 2018; Raychev et al., 2016; Iyer et al., 2018; Lu et al., 2021) main paper.”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, positive): “We use the same dataset released by Iyer et al. (2018), which consists of 100k training samples, 2k validation samples and 2k test samples.”
Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing (2019, positive): “Recently, context-dependent semantic parsing has drawn plenty of attention (Long et al., 2016; Iyyer et al., 2017; Iyer et al., 2018; Suhr et al., 2018; Suhr and Artzi, 2018), where the generation of logical forms is conditioned on the context environment.”
UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022, unknown): “we use CONCODE (Iyer et al., 2018) dataset, where the input consists of an NL description and code environments.”
In-IDE Code Generation from Natural Language: Promise and Challenges (2021, unknown): “Specifically, many code generation methods [1, 47, 115, 120] aim to directly compare generated code snippets against ground truth snippets, using token sequence comparison metrics borrowed from machine translation tasks, such as BLEU score [86].”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, unknown): “LLMs have demonstrated state-ofthe-art performance on various code-related tasks, including code generation/synthesis [11, 28, 24], program repair [58, 59, 57], code translation [27, 45] and code summarization [1, 30].”
Execution-based Evaluation for Data Science Code Generation Models (2022, unknown): “…natural language (Wang et al., 2015; Oda et al., 2015; Zhong et al., 2017; Yin et al., 2018; Yu et al., 2018; Lin et al., 2018), unﬁnished code (Iyer et al., Consequently, recent works turn to execution-based metrics instead, where the code would be correct if it passes a set of unit tests…”
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, unknown): “Statement-level code generation (Iyer et al., 2018; Athiwaratkun et al., 2022) aims to output statement-specific source codes.”
A Survey on Large Language Models for Code Generation (2024, unknown): “…enhance its system for identifying and extracting meaningful information from text (e.g. names, locations, and dates), allowing Microsoft to enhance language understanding, deliver reliable, context-aware information, and improve applications in NLP, search engines, and information retrieval [14].”
Structural Language Models for Any-Code Generation (2019, unknown): “Node Trees vs. Production Trees While we predict a single node at each step, previous work (Iyer et al., 2018; Brockschmidt et al., 2019a) predicts a grammar production rule.”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, unknown): “However, CONCODE (Iyer et al., 2018) is the only language-to-code task included in CodeXGLUE and it uses surface-form based evaluation metrics such as BLEU.”
SPoC: Search-based Pseudocode to Code (2019, unknown): “…text descriptions to a one-line program [42, 35, 43, 44, 21, 8], and while recent work explored generating longer programs from text descriptions [22, 40, 29, 15, 16, 14], these programs are mostly evaluated on syntactic metrics (e.g., exact match and BLEU score) rather than functional correctness.”