Predicting Program Properties from "Big Code"
By Veselin Raychev, Martin T. Vechev, Andreas Krause. 2015
405 citations recorded by the source corpus.
We present a new approach for predicting program properties from massive codebases (aka "Big Code"). Our approach first learns a probabilistic model from existing data and then uses this model to predict properties of new, unseen programs. The key idea of our work is to transform the input program into a representation which allows us to phrase the problem of inferring program properties as structured prediction in machine learning. This formulation enables us to leverage powerful probabilistic graphical models such as conditional random fields (CRFs) in order to perform joint prediction of program properties. As an example of our approach, we built a scalable prediction engine called JSNice for solving two kinds of problems in the context of JavaScript: predicting (syntactic) names of identifiers and predicting (semantic) type annotations of variables. Experimentally, JSNice predicts correct names for 63% of name identifiers and its type annotation predictions are correct in 81% of the cases. In the first week since its release, JSNice was used by more than 30,000 developers and in only few months has become a popular tool in the JavaScript developer community. By formulating the problem of inferring program properties as structured prediction and showing how to perform both learning and inference in this context, our work opens up new possibilities for attacking a wide range of difficult problems in the context of "Big Code" including invariant generation, decompilation, synthesis and others.
Subfields: Information Systems, Software, Code Generation & Program Synthesis, Program Analysis & Verification, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Miltiadis Allamanis
- Charles Sutton
- Michael Pradel
- Uri Alon
- Eran Yahav
- Premkumar T. Devanbu
- Rishabh Singh
- Earl T. Barr
In Software
- Charles Sutton
- Miltiadis Allamanis
- Michael Pradel
- Premkumar T. Devanbu
- Rishabh Singh
- Koushik Sen
- Earl T. Barr
- Marc Brockschmidt
In Code Generation & Program Synthesis
- Charles Sutton
- Miltiadis Allamanis
- Michael Pradel
- Premkumar T. Devanbu
- Rishabh Singh
- Vincent J. Hellendoorn
- Rafael-Michael Karampatsis
- Swarat Chaudhuri
In Program Analysis & Verification
- Miltiadis Allamanis
- Charles Sutton
- Michael Pradel
- Premkumar T. Devanbu
- Marc Brockschmidt
- Earl T. Barr
- Georgios Gousios
- Rafael-Michael Karampatsis
In Signal Processing
- Miltiadis Allamanis
- Charles Sutton
- Michael Pradel
- Koushik Sen
- Eran Yahav
- Marc Brockschmidt
- Uri Alon
- Premkumar T. Devanbu
Citations in the corpus
- Program Synthesis with Large Language Models (2021, methodology): “Methods aim improving code readability by asking a model trained on a code corpus with good style to predict names of variables and methods in new code [Raychev et al., 2015, Allamanis et al., 2014, Alon et al., 2019].”
- The adverse effects of code duplication in machine learning models of code (2018, methodology): “Task: Variable Naming Model: JsNice [23] Dataset : Reshuffled & Reduced JavaScript-150k [22] Accuracy (%) 34.”
- A Survey of Machine Learning for Big Code and Naturalness (2017, methodology): “[165] use a graphical model to predict the probability distribution of JavaScript variable types by learning statistical patterns of how code is used.”
- Neuro-Symbolic Program Synthesis (2016, background): “There is also a recent line of work on learning probabilistic models of code from a large number of code repositories ( big code ) (Raychev et al., 2015; Bielik et al., 2016; Hindle et al., 2016), which are then used for applications such as auto-completion of partial programs, inference of…”
- code2seq: Generating Sequences from Structured Representations of Code (2018, background): “…syntactic information have usually been evaluated on relatively easier tasks, which mainly focus on “filling the blanks” in a given program (Alon et al., 2018b; Bielik et al., 2016; Raychev et al., 2016; 2015; Allamanis et al., 2018) or semantic classification of code snippets (Alon et al., 2018a).”
- Learning to Represent Programs with Graphs (2017, methodology): “Raychev et al. (2015) and Bichsel et al. (2016) use conditional random fields to model a variety of relationships between variables, AST elements and types to predict variable names and types (resp. to deobfuscate Android apps), but without considering the flow of data explicitly.”
- A Convolutional Attention Network for Extreme Summarization of Source Code (2016, methodology): “Rush et al.(2015) use an attention-based encoder to summarize sentences, but do not use convolution for their attention mechanism.”
- Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, background): “Second, source code is available in large amounts, thanks to the proliferation of open source software in general, and the popularity of open access, “Big Code” repositories like GitHub and Stack Overflow (SO); these platforms host tens of millions of code repositories and programming-related questi”
- DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “UnuglifyJS1, which is based on JSNICE (Raychev et al., 2015) and available online, is especially famous in the Javascript community.”
- DeepBugs: a learning approach to name-based bug detection (2018, methodology): “As observed by others [Bielik et al. 2016; Nguyen and Nguyen 2015; Raychev et al. 2015], the huge amount of existing code provides ample of examples of likely correct code.”
- LambdaNet: Probabilistic Type Inference using Graph Neural Networks (2020, methodology): “While there has been some prior work on inferring type annotations for TypeScript using machine learning (Hellendoorn et al., 2018; Raychev et al., 2015), prior work in this space has several shortcomings.”
- Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts (2017, methodology): “f training and testing files to 64,750 and 33,229, respectively. A possible alternative to using fixed training and testing sets is k-fold cross validation. We follow the experimental setup of JSNice [31] and JSNaughty [38], which uses fixed training and testing sets. We present some statistics abou”
- Modeling Vocabulary for Big Code Machine Learning (2019, methodology): “They then learn a probabilistic model from existing data and use it to predict properties (e.g., variable names or type annotations) of unseen programs et al. [59] .”
- Bayesian specification learning for finding API usage errors (2017, methodology): “JSNice [46] uses a probabilistic graphical model to learn lexical and syntactical properties of programs such as variable names and types for the purpose of de-obfuscating Javascript programs.”
- Recovering clear, natural identifiers from obfuscated JS names (2017, methodology): “Our tool, JSnaughty, realizes this intuition via two complementary mechanisms: a) the existing JSnice tool [42], which conditions “natural” names on semantic properties of the mini ed code, using static analysis; and b) Autonym, which uses an o -the-shelf statistical machine translation tool, and is”
- Neuro-Symbolic Program Corrector for Introductory Programming Assignments (2018, methodology): “Our approach is inspired from the recent pioneering work on learning probabilistic models of source code from a large repository of code for many different applications [1–4, 13, 19, 24].”