A Survey of Machine Learning for Big Code and Naturalness

By Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, Charles Sutton. 2017

1,037 citations recorded by the source corpus.

Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit the abundance of patterns of code. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.

Subfields: Information Systems, Software, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Prior work exists on evaluating readability (Allamanis et al., 2018) and security (Pearce et al.”
TypeWriter: neural type prediction with search-based validation (2019, methodology): “Our neural type prediction model is motivated by a stream of work on machine learning-based program analyses [2].”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, background): “Applying machine learning to code has been widely considered [2].”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, background): “As code artificial intelligence receives more and more attention (Allamanis et al. 2015; Yin and Neubig 2017; Allamanis et al. 2018; Monperrus 2018; Alon et al. 2019; Svyatkovskiy et al. 2020), the evaluation of code synthesis becomes critical to promote its development.”
Learning to Represent Programs with Graphs (2017, methodology): “Our work builds upon the recent ﬁeld of using machine learning for source code artifacts (Allamanis et al., 2017).”
code2vec: learning distributed representations of code (2018, background): “2016], and more (see [Allamanis et al. 2017; Vechev and Yahav 2016] for a survey).”
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair (2018, background): “’s [8] for the latter.”
Tree-to-tree Neural Networks for Program Translation (2018, methodology): “Some recent work have applied statistical machine translation techniques to program translation [2, 16, 22, 21, 23, 24].”
DeepBugs: a learning approach to name-based bug detection (2018, background): “A recent survey discusses more approaches [Allamanis et al. 2017a].”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “Many works have taken advantage of the “naturalness” of software [44] to assist software engineering tasks, including code completion [76], improving code readability [2], program repair [20, 78], identifying buggy code [75] and API migration [38], among many others [4].”
DeepDelta: learning to repair compilation errors (2019, methodology): “This has come to be known as the software naturalness hypothesis [2], which states that large code corpora are statistically similar to natural-language text, since coding is also an act of human communication.”
Deep Learning to Find Bugs (2017, neutral): “A recent survey discusses more approaches [3].”
Typilus: neural type hints (2020, background): “Recently, researchers have been exploring the application of machine learning to code [4].”
CoCoNuT: combining context-aware neural translation models using ensemble for program repair (2020, methodology): “Deep Learning in Software Engineering: The software engineering community had applied deep learning to perform various tasks such as defects prediction [40, 79, 81], source code representation [6, 7, 66, 80], source code summarization [4, 22] source code modeling [5, 11, 26, 83], code clone detectio”
Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, neutral): “There is a growing trend towards using AI for building tool support for code [1][2].”