Deep Learning Based Vulnerability Detection: Are We There Yet?
By Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, Baishakhi Ray. 2020
789 citations recorded by the source corpus.
Automated detection of software vulnerabilities is a fundamental problem in software security. Existing program analysis techniques either suffer from high false positives or false negatives. Recent progress in Deep Learning (DL) has resulted in a surge of interest in applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy of up to 95 percent at detecting vulnerabilities. In this paper, we ask, “how well do the state-of-the-art DL-based techniques perform in a real-world vulnerability prediction scenario?” To our surprise, we find that their performance drops by more than 50 percent. A systematic investigation of what causes such precipitous performance drop reveals that existing DL-based vulnerability prediction approaches suffer from challenges with the training data (e.g., data duplication, unrealistic distribution of vulnerable classes, etc.) and with the model choices (e.g., simple token-based models). As a result, these approaches often do not learn features related to the actual cause of the vulnerabilities. Instead, they learn unrelated artifacts from the dataset (e.g., specific variable/function names, etc.). Leveraging these empirical findings, we demonstrate how a more principled approach to data collection and model design, based on realistic settings of vulnerability prediction, can lead to better solutions. The resulting tools perform significantly better than the studied baseline—up to 33.57 percent boost in precision and 128.38 percent boost in recall compared to the best performing model in the literature. Overall, this paper elucidates existing DL-based vulnerability prediction systems’ potential issues and draws a roadmap for future DL-based vulnerability prediction research.
Subfields: Information Systems, Software, Program Analysis & Verification, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Shaohua Wang
- Yi Li
- Tien Nhut Nguyen
- Xinyun Chen
- Baishakhi Ray
- Saikat Chakraborty
- Chakkrit Kla Tantithamthavorn
- Michael Fu
In Software
- Shaohua Wang
- Yi Li
- Tien Nhut Nguyen
- Chakkrit Kla Tantithamthavorn
- Michael Fu
- Baishakhi Ray
- Wasi Uddin Ahmad
- Xinyun Chen
In Program Analysis & Verification
- Tien Nhut Nguyen
- Shaohua Wang
- Yi Li
- Chakkrit Kla Tantithamthavorn
- Michael Fu
- Xinyun Chen
- Baishakhi Ray
- Yizheng Chen
In Privacy & Security
Citations in the corpus
- Unified Pre-training for Program Understanding and Generation (2021, background): “…et al., 2018; Harer et al., 2019; Ahmad et al., 2020), Bug Detection (Ray et al., 2016; Li et al., 2018b; Russell et al., 2018; Zhou et al., 2019; Chakraborty et al., 2020), Program Repair (Chen et al., 2019; Chakraborty et al., 2020; Lutellier et al., 2020), Code Translation (Chen et al., 2018;…”
- Vulnerability detection with fine-grained interpretations (2021, methodology): “5) Reveal [11]: this approach uses GGNN, MLP, and with Triplet Loss on graph representations of source code.”
- LineVul: A Transformer-based Line-Level Vulnerability Prediction (2022, methodology): “Chakraborty et al. [12] proposed Reveal, which is a Gated Graph Neural Network (GGNN) that learns the graph properties of source code.”
- DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection (2023, methodology): “In previous papers, researchers believe that GNN with manual feature engineering is promising for vulnerability detection [4, 17, 31], since it combines domain knowledge with deep learning.”