DeepBugs: a learning approach to name-based bug detection
By Michael Pradel, Koushik Sen. 2018
396 citations recorded by the source corpus.
Natural language elements in source code, e.g., the names of variables and functions, convey useful information. However, most existing bug detection tools ignore this information and therefore miss some classes of bugs. The few existing name-based bug detection approaches reason about names on a syntactic level and rely on manually designed and tuned algorithms to detect bugs. This paper presents DeepBugs, a learning approach to name-based bug detection, which reasons about names based on a semantic representation and which automatically learns bug detectors instead of manually writing them. We formulate bug detection as a binary classification problem and train a classifier that distinguishes correct from incorrect code. To address the challenge that effectively learning a bug detector requires examples of both correct and incorrect code, we create likely incorrect code examples from an existing corpus of code through simple code transformations. A novel insight learned from our work is that learning from artificially seeded bugs yields bug detectors that are effective at finding bugs in real-world code. We implement our idea into a framework for learning-based and name-based bug detection. Three bug detectors built on top of the framework detect accidentally swapped function arguments, incorrect binary operators, and incorrect operands in binary operations. Applying the approach to a corpus of 150,000 JavaScript files yields bug detectors that have a high accuracy (between 89% and 95%), are very efficient (less than 20 milliseconds per analyzed file), and reveal 102 programming mistakes (with 68% true positive rate) in real-world code.
Subfields: Information Systems, Software, Natural Language Processing, Program Analysis & Verification, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Charles Sutton
- Michael Pradel
- Georgios Gousios
- Rishabh Singh
- Petros Maniatis
- Percy Liang
- Satish Chandra
- Miltiadis Allamanis
In Software
- Michael Pradel
- Charles Sutton
- Rishabh Singh
- Percy Liang
- Michihiro Yasunaga
- Tien Nhut Nguyen
- Rafael-Michael Karampatsis
- Jibesh Patra
In Natural Language Processing
- Charles Sutton
- Percy Liang
- Michael Pradel
- Rishabh Singh
- Uri Alon
- Eran Yahav
- Miltiadis Allamanis
- Michihiro Yasunaga
In Program Analysis & Verification
- Petros Maniatis
- Aditya Kanade
- Georgios Gousios
- Percy Liang
- Michihiro Yasunaga
- Rishabh Singh
- Shaohua Wang
- Yi Li
In Signal Processing
- Eran Yahav
- Michael Pradel
- Uri Alon
- Charles Sutton
- Georgios Gousios
- Rafael-Michael Karampatsis
- Jibesh Patra
- Satish Chandra
Citations in the corpus
- Program Synthesis with Large Language Models (2021, methodology): “Many learning methods have been proposed for program repair [Allamanis et al., 2018b, Tarlow et al., 2019, Hellendoorn et al., 2019, Dinella et al., 2019, Yasunaga and Liang, 2020, Chen et al., 2019b, Pradel and Sen, 2018].”
- Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “As labeled data is costly to prepare, various works study learning code fixers from unlabeled or synthetic data (Pu et al., 2016; Parihar et al., 2017; Ahmed et al., 2018; Pradel & Sen, 2018; Wang et al., 2018; Vasic et al., 2019; Gupta et al., 2019; Hajipour et al., 2019; Hellendoorn et al., 2020).”
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “Finally, several works focus on repairing specific types of bugs, e.g., variable misuse (Vasic et al., 2019), name-based bugs (Pradel & Sen, 2018) and Javascript bugs (Dinella et al., 2020).”
- TypeWriter: neural type prediction with search-based validation (2019, methodology): “ype prediction model is motivated by a stream of work on machine learning-based program analyses [2]. Beyond type prediction, others have proposed learning-based techniques to find programming errors [25, 27], predict variable and method names [1, 28, 33], suggest how to improve names [19], search c”
- Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “Similarly, Pradel and Sen [70] had a large vocabulary of 2.4 million unique tokens: they limited it to the 10,000 most common tokens to reduce inaccuracies due to rare words.”
- Neural Program Repair by Jointly Learning to Localize and Repair (2019, background): “Unlike DeepBugs, which embeds a single expression, our model embeds the full input program (up to a maximum prefix length) and performs both localization and repair of the V AR M ISUSE bugs in addition to the classification task.”
- Adversarial examples for models of code (2019, background): “Neural models of code have recently achieved state-of-theart performance on various tasks such as prediction of variable names and types [1, 5, 10, 32], code summarization [2, 3, 18], code generation [4, 11, 28], code search [13, 25, 34], and bug finding [29, 33, 35].”
- Learning and Evaluating Contextual Embedding of Source Code (2019, background): “On the Operand task, the performance difference between CuBERT with 2 or 10 fine-tuning epochs and BiLSTM is about 1%.”
- Getafix: learning to fix bugs automatically (2019, neutral): “t could for example decide between early return patterns and wrapping patterns. We have already experimented with training a classifier, hoping to be more sensitive to such subtleties. Pradel and Sen [15] showed promising results in purely name-based bug detection. It is likely that using a similar ”
- NL2Type: Inferring JavaScript Function Types from Natural Language Information (2019, methodology): “Finally, DeepBugs uses natural language information in code, in particular identifier names, to detect code that is likely to be incorrect [28].”
- Improving bug detection via context-based code representation learning and attention-based neural networks (2019, methodology): “• Machine learning-based bug detection [Pradel and Sen 2018; Wang et al. 2016a,b; Wasylkowski et al. 2007].”
- Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, background): “…could have broad application throughout software engineering, such as in models to suggest readable function and class names [3], summarizing source code [5, 43], predicting bugs [60], detecting code clones [73], comment generation [42], fixing syntactic errors [47], and variable de-obfuscation [9].”
- An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
- A Novel Neural Source Code Representation Based on Abstract Syntax Tree (2019, background): “DeepBugs [61] represents code via word2vec for detecting name-based bugs.”
- MISIM: An End-to-End Neural Code Similarity System (2020, background): “Accurate code similarity systems may assist in many programming tasks, such as code recommendation systems to improve programming development productivity, to automated bug detection and mitigation systems to improve programmer debugging productivity [2, 6, 8, 11, 14, 29, 34].”
- Pre-trained Contextual Embedding of Source Code (2019, background): “Swapped Operand Pradel & Sen (2018) propose the wrong binary operand task where a variable or constant is used incorrectly in an expression, but that task is quite similar to the variable misuse task we already use.”