EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
By H. S. Anderson, Phil Roth. arXiv.org, 2018
657 citations recorded by the source corpus.
This paper describes EMBER: a labeled benchmark dataset for training machine learning models to statically detect malicious Windows portable executable files. The dataset includes features extracted from 1.1M binary files: 900K training samples (300K malicious, 300K benign, 300K unlabeled) and 200K test samples (100K malicious, 100K benign). To accompany the dataset, we also release open source code for extracting features from additional binaries so that additional sample features can be appended to the dataset. This dataset fills a void in the information security machine learning community: a benign/malicious dataset that is large, open and general enough to cover several interesting use cases. We enumerate several use cases that we considered when structuring the dataset. Additionally, we demonstrate one use case wherein we compare a baseline gradient boosted decision tree model trained using LightGBM with default settings to MalConv, a recently published end-to-end (featureless) deep learning model for malware detection. Results show that even without hyper-parameter optimization, the baseline EMBER model outperforms MalConv. The authors hope that the dataset, code and baseline model provided by EMBER will help invigorate machine learning research for malware detection, in much the same way that benchmark datasets have advanced computer vision research.
Subfields: Artificial Intelligence, Computer Networks and Communications, Signal Processing
View Paper: https://www.semanticscholar.org/paper/7e152e587fbc73b5bd23048c71c1b36c569416c5
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Giorgio Severi
- Ömer Aslan
- Refik Samet
- Ernesto Damiani
- J. Meyer
- Zhibo Zhang
- Fatma Taher
- Chan Yeob Yeun
In Computer Networks and Communications
- Giorgio Severi
- Ömer Aslan
- Refik Samet
- Ernesto Damiani
- J. Meyer
- Zhibo Zhang
- Fatma Taher
- Chan Yeob Yeun
In Signal Processing
- Giorgio Severi
- Ömer Aslan
- Refik Samet
- Ernesto Damiani
- J. Meyer
- Zhibo Zhang
- Fatma Taher
- Chan Yeob Yeun
Citations in the corpus
- Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, methodology): “The gradients and weights of different layers of the MalConv architecture [155] and emberMalConv [156] are analyzed to identify different parts’ contributions to the classification.”
- A Comprehensive Review on Malware Detection Approaches (2020, background): “• EMBER dataset (2018): It consists of 1 million records and holds malware and benign features [27].”
- Robust Intelligent Malware Detection Using Deep Learning (2019, methodology): “In addition, we compare the performance of various classical machine learning classifiers on the domain level features obtained from [26] using deep learning techniques.”
- Exploring Backdoor Poisoning Attacks Against Malware Classifiers (2020, methodology): “We implement a watermarking utility and apply the selected watermarks to real goodware and malware samples from the EMBER dataset.”