Intriguing Properties of Adversarial ML Attacks in the Problem Space

By Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, Lorenzo Cavallaro. 2019

358 citations recorded by the source corpus.

Recent research efforts on adversarial ML have investigated problem-space attacks, focusing on the generation of real evasive objects in domains where, unlike images, there is no clear inverse mapping to the feature space (e.g., software). However, the design, comparison, and real-world implications of problem-space attacks remain underexplored.This paper makes two major contributions. First, we propose a novel formalization for adversarial ML evasion attacks in the problem-space, which includes the definition of a comprehensive set of constraints on available transformations, preserved semantics, robustness to preprocessing, and plausibility. We shed light on the relationship between feature space and problem space, and we introduce the concept of side-effect features as the byproduct of the inverse feature-mapping problem. This enables us to define and prove necessary and sufficient conditions for the existence of problem-space attacks. We further demonstrate the expressive power of our formalization by using it to describe several attacks from related literature across different domains.Second, building on our formalization, we propose a novel problem-space attack on Android malware that overcomes past limitations. Experiments on a dataset with 170K Android apps from 2017 and 2018 show the practical feasibility of evading a state-of-the-art malware classifier along with its hardened version. Our results demonstrate that "adversarial-malware as a service" is a realistic threat, as we automatically generate thousands of realistic and inconspicuous adversarial applications at scale, where on average it takes only a few minutes to generate an adversarial app. Yet, out of the 1600+ papers on adversarial ML published in the past six years, roughly 40 focus on malware [15]—and many remain only in the feature space.Our formalization of problem-space attacks paves the way to more principled research in this domain. We responsibly release the code and dataset of our novel attack to other researchers, to encourage future work on defenses in the problem space.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “Prior research on the security of neural code models focused on code summarization and classification (especially for malware analysis [24, 44]), only in the setting where the attacker can modify inputs into the model at inference time.”
Unsolved Problems in ML Safety (2021, background): “[138] Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro.”
Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey (2023, background): “Moreover, it is known that the packet mutations can produce side effect features [121] ( e.g., injecting new packets into the trafﬁc or fragmenting packets can cause new features to be extracted), but existing packet-level adversarial attacks have largely ignored the impact of side effect features.”
AdVersa: Adversarially-Robust and Practical Ad and Tracker Blocking in the Wild (2026, unknown): “This aligns with the “inverse feature-mapping” challenge identified in priorwork[46],whichhighlightsthedifficultyoftranslatingfeature-spaceperturbationsintovalidproblem-spaceobjects.”