LEMNA: Explaining Deep Learning based Security Applications

By Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, G. Wang, Xinyu Xing. 2018

348 citations recorded by the source corpus.

While deep learning has shown a great potential in various domains, the lack of transparency has limited its application in security or safety-critical areas. Existing research has attempted to develop explanation techniques to provide interpretable explanations for each classification decision. Unfortunately, current methods are optimized for non-security tasks ( e.g., image analysis). Their key assumptions are often violated in security applications, leading to a poor explanation fidelity. In this paper, we propose LEMNA, a high-fidelity explanation method dedicated for security applications. Given an input data sample, LEMNA generates a small set of interpretable features to explain how the input sample is classified. The core idea is to approximate a local area of the complex deep learning decision boundary using a simple interpretable model. The local interpretable model is specially designed to (1) handle feature dependency to better work with security applications ( e.g., binary code analysis); and (2) handle nonlinear local boundaries to boost explanation fidelity. We evaluate our system using two popular deep learning applications in security (a malware classifier, and a function start detector for binary reverse-engineering). Extensive evaluations show that LEMNA's explanation has a much higher fidelity level compared to existing methods. In addition, we demonstrate practical use cases of LEMNA to help machine learning developers to validate model behavior, troubleshoot classification errors, and automatically patch the errors of the target models.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Despite intense interest and collective group efforts, we are only seeing limited progress in definitions [8], frameworks [9], visualization [10], and limited experimentation [11].”
TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems (2019, methodology): “resent a high-fidelity restoration. To address this problem and improve fidelity for resolved triggers, we design a regularization term by borrowing the idea of those explainable AI techniques (e.g., [2, 10, 14, 19, 20, 35, 37, 39, 44]). The rationale behind our design is as follows. An overlaying t”
Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey (2019, background): “These models report only the detection results and have no interpretable basis [93].”
Interpretable Deep Learning under Fire (2018, background): “Such interpretability helps users understand the inner workings of DNNs, enabling use cases including model debugging [39], digesting security analysis results [20], and detecting adversarial inputs [13].”
Parameterized Explainer for Graph Neural Network (2020, background): “Methods demystifying complicated deep learning models can be grouped into two main families, whitebox and blackbox [19, 20].”
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning (2019, neutral): “Besides the above, there exist a wide range of other attacks and defenses on machine learning models [4,5,8,9,13,14,16, 17, 22, 23, 25, 26, 35, 41, 43, 44, 46, 48–50, 52, 54–56]”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, methodology): “utput layer but only several front layers, we could try to apply some Explainable Machine Learning techniques to extract the key features of each class, like LIME [42] for linear classiﬁers and LEMNA [43] for deep neural networks. Explainable Machine Learning aims at ﬁguring out why a particular inp”
A Survey of Android Malware Detection with Deep Neural Models (2020, background): “Thus, the community researchers should pay more attention to the model transparency and interpretation for malware detection [142].”
Deep Learning Based Vulnerability Detection: Are We There Yet? (2020, methodology): “Using state-of-the-art explanation techniques [9], we find that the current models are essentially learning up irrelevant features that are not related to vulnerabilities and are likely”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “Attacks against such ML models can be targeted towards compromising the integrity of the model (such as by evasion attacks [5,13,18,32,63,76,79,84,87]), or its privacy and con-ﬁdentiality .”
TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (2018, methodology): “Instead, we demonstrate how, in this specific setting, TESSERACT allowed us to highlight higher robustness of DL [22] against time decay; however, the prices to pay to use DL are lower explainability [23, 42] and higher training time [19].”