Structured Generative Models of Natural Source Code

By Chris J. Maddison, Daniel Tarlow. 2014

175 citations recorded by the source corpus.

We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “Maddison & Tarlow (2014) improved on this setup by learning a state vector used to condition child node expansion.”
The adverse effects of code duplication in machine learning models of code (2018, methodology): “We also compute focused metrics on identifiers since they have been proven to be the hardest to predict [4, 9, 20].”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “t they do not even capture close-range dependencies that matter [29]. Further, ASTs tend to be deeper and wider than text parse trees due to the highly compositional nature of code. Maddison andTarlow[126]andAllamaniset al.[13]increase the size ofthecontext considered by creating a non-context-free ”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, background): “Code generation is well studied [21, 17, 4, 23, 1], but these approaches have not explored edit-based generation.”
Learning to Execute (2014, background): “N E ] 17 O c t 2014 (Maddison & Tarlow, 2014) learned a language model on parse trees, and (Mou et al., 2014) predicted whether two programs are equivalent or not.”
Graphite: Iterative Generative Modeling of Graphs (2018, background): “Finally, there has been a fair share of recent work for generation of special kinds of graphs, such as parsed trees of source code [29] and SMILES representations for molecules [33].”
Neuro-Symbolic Program Synthesis (2016, background): “The work by Maddison & Tarlow (2014) addresses the problem of learning structured generative models of source code but both their model and application domain are different from ours.”
Learning Deep Generative Models of Graphs (2018, background): “Maddison & Tarlow (2014) developed probabilistic models of parsed syntax trees for source code.”
Learning to Represent Programs with Graphs (2017, methodology): “For example, Hindle et al. (2012); Bhoopchand et al. (2016) model the code as a sequence of tokens, while Maddison & Tarlow (2014); Raychev et al. (2016) model the syntax tree structure of code.”
A Convolutional Attention Network for Extreme Summarization of Source Code (2016, background): “Research has mostly focused on token-level (Nguyen et al., 2013; Tu et al., 2014) and syntax-level (Maddison & Tarlow, 2014) language models of code and translation between programming languages (Karaivanov et al., 2014; Nguyen et al., 2014).”
code2vec: learning distributed representations of code (2018, background): “Allamanis et al. [2015b]; Maddison and Tarlow [2014] also addressed the problem of translating between code and natural language, by considering the syntax of the code rather than representing it as a tokens stream.”
Learning to Fix Build Errors with Graph2Diff Neural Networks (2019, background): “Graph2Tocopo is a single abstraction that unifies ideas from many recent works in modelling source code [3, 4, 8, 22, 35, 44].”
Probabilistic model for code with decision trees (2016, background): “Another recent work proposes a language model of C# programs based on a log-linear tree-traversal model [19].”
Learning API Usages from Bytecode: A Statistical Approach (2015, background): “[25] proposed tree-based generative models for source code.”
PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair (2021, background): “…led to a wealth of literature going beyond text-to-text models that make use of graphs, copy mechanisms, pointers, and other custom architectures [Maddison and Tarlow, 2014, Bielik et al., 2016, Allamanis et al., 2016, Mukherjee et al., 2017, Yin and Neubig, 2017, Allamanis et al., 2018, Alon et…”
Learning to Represent Edits (2018, background): “Modeling tree generation with machine learning is an old problem that has been widely studied in NLP. Starting with Maddison & Tarlow (2014), code generation has also been considered as a tree generation problem.”