On the "Naturalness" of Buggy Code

By Baishakhi Ray, Vincent J. Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, Premkumar T. Devanbu. 2015

299 citations recorded by the source corpus.

Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be “natural”, like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical language model is “unnatural” in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca. 7,139), from 10 different Java projects, and focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e. unnatural), becoming less so as bugs are fixed. Ordering files for inspection by their average entropy yields cost-effectiveness scores comparable to popular defect prediction methods. At a finer granularity, focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and or- dering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid, simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.

Subfields: Information Systems, Software, Code Generation & Program Synthesis, Program Analysis & Verification, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Code Generation & Program Synthesis

In Program Analysis & Verification

In Signal Processing

Citations in the corpus

A Survey of Machine Learning for Big Code and Naturalness (2017, background): “[163] PD (c) Token n-gram (cache) Bug Detection”
Unified Pre-training for Program Understanding and Generation (2021, background): “, 2020), Bug Detection (Ray et al., 2016; Li et al., 2018b; Russell et al., 2018; Zhou et al., 2019; Chakraborty et al., 2020), Program Repair (Chen et al.”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, background): “• code-code (clone detection [65, 84, 46, 80, 9, 89, 86], defect detection [91, 55, 51, 42, 78, 79], cloze test [15], code completion [57, 1, 56, 66, 67, 8, 27, 7], code repair [69, 4, 23, 26, 71, 73], and code-to-code translation [48, 37, 10, 41]) • text-code (natural language code search [32, 20, ”
DeepBugs: a learning approach to name-based bug detection (2018, background): “Even though it has been noted that buggy code stands out compared to non-buggy code [Ray et al. 2016], little work exists on automatically detecting bugs via machine learning.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, methodology): “Ray et al. [75] showed that buggy code has on average lower probability than correct code, and that LMs can spot defects as eﬀectively as popular tools such as FindBugs.”
Modeling Vocabulary for Big Code Machine Learning (2019, background): “vantage of the "naturalness" of software [33] to assist a variety of software engineering tasks, including code completion [60], correcting syntax errors [62], detecting possibly buggy code [58], or API migration [53], among many others [4]. These approaches analyze large amounts of source”
Bugram: Bug detection with n-gram language models (2016, methodology): “It has recently been demonstrated that n-gram language models [7] can capture the regularities of software source code [15,36].”
When Code Completion Fails: A Case Study on Real-World Completions (2019, background): “Numerous studies have developed tools that can assist programmers in common, but taxing tasks, such as fault localization, program patching, type annotation, and API recommendation [1]–[4].”
Graph-Based Mining of In-the-Wild, Fine-Grained, Semantic Code Change Patterns (2019, methodology): “Several researchers have exploited the repetitive nature of code changes [3], [4], [5], [6], [16], [19], [26], [27], [31], [32], [36], [39], [44], [46], [47].”
Improving bug detection via context-based code representation learning and attention-based neural networks (2019, methodology): “We use the same approach as the ones used in [Mockus and Votta 2000; Ray et al. 2016, 2014] to process each commit message and mark it as a bug-fix if the message contains a bug id and at least one of the error related keywords: {error, bug, fix, issue, mistake, incorrect, fault, defect, flaw and…”
A Novel Neural Source Code Representation Based on Abstract Syntax Tree (2019, background): “Such a vector captures the naturalness of source code [32], [33] and can serve as a neural source code representation.”
Neuro-Symbolic Program Corrector for Introductory Programming Assignments (2018, methodology): “Ray et al. [23] recently showed that even syntactically correct buggy code are typically unnatural, which can then be used to assist bug repair methods.”
Deep learning code fragments for code clone detection (2016, methodology): “To the best of our knowledge, we are the ﬁrst to propose language models and embeddings for clone detection.”
Are deep neural networks the best choice for modeling source code? (2017, methodology): “As we shall see, performance in this final setting is best-in-class, and has ramifications for code synthesis [14, 30], fault localization [8, 31], code naturalization [2] and many real-world code suggestions settings [15, 29].”
CODIT: Code Editing With Tree-Based Neural Models (2018, methodology): “Applying ML to source code has received increasing attention in recent years [26] across many applications such as code completion [11], [31], bug prediction [57], [58], [59], clone detection [60], code search [61], etc.”
Automated program repair (2019, background): “This can often work by virtue of the fact that source code is repetitive [14].”
Recovering clear, natural identifiers from obfuscated JS names (2017, background): “There have been substantial further applications of statistical models for code, in areas such as coding standards mining and checking [1], code summarization [17], idiom mining [3, 37], and bug localization [41].”