Reviewer Integration and Performance Measurement for Malware Detection

By Brad Miller, Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Rekha Bachwani, Riyaz Faizullabhoy, Ling Huang, Vaishaal Shankar, Tony Wu, George Yiu and 2 more. 2015

89 citations recorded by the source corpus.

We present and evaluate a large-scale malware detection system integrating machine learning with expert reviewers, treating reviewers as a limited labeling resource. We demonstrate that even in small numbers, reviewers can vastly improve the system's ability to keep pace with evolving threats. We conduct our evaluation on a sample of VirusTotal submissions spanning 2.5i?źyears and containing 1.1 million binaries with 778i?źGB of raw feature data. Without reviewer assistance, we achieve 72i?ź% detection at a 0.5i?ź% false positive rate, performing comparable to the best vendors on VirusTotal. Given a budget of 80 accurate reviews daily, we improve detection to 89i?ź% and are able to detect 42i?ź% of malicious binaries undetected upon initial submission to VirusTotal. Additionally, we identify a previously unnoticed temporal inconsistency in the labeling of training datasets. We compare the impact of training labels obtained at the same time training data is first seen with training labels obtained months later. We find that using training labels obtained well after samples appear, and thus unavailable in practice for current training data, inflates measured detection by almost 20i?ź% points. We release our cluster-based implementation, as well as a list of all hashes in our evaluation and 3i?ź% of our entire dataset.

Subfields: Computer Networks and Communications, Information Systems, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Signal Processing

Citations in the corpus

TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (2018, methodology): “[29] found that AVs sometimes change their outcome over time; in particular, some goodware may eventually be tagged as malware.”
Intriguing Properties of Adversarial ML Attacks in the Problem Space (2019, background): “[49]): an app is considered goodware if it has 0 VirusTotal detections, as malware if it has 4+ VirusTotal detections, and is discarded as grayware if it has between 1 and 3 VirusTotal detections.”
AVclass: A Tool for Massive Malware Labeling (2016, background): “Recently, Miller et al. [19] argue that detection systems should be trained not with the labels available at evaluation time (e.g., latest VT reports), but with the labels available at training time.”