Security analysis of online centroid anomaly detection

By Marius Kloft, Pavel Laskov. 2010

97 citations recorded by the source corpus.

Security issues are crucial in a number of machine learning applications, especially in scenarios dealing with human activity rather than natural phenomena (e.g., information ranking, spam detection, malware detection, etc.). In such cases, learning algorithms may have to cope with manipulated data aimed at hampering decision making. Although some previous work addressed the issue of handling malicious data in the context of supervised learning, very little is known about the behavior of anomaly detection methods in such scenarios. In this contribution, we analyze the performance of a particular method--online centroid anomaly detection--in the presence of adversarial noise. Our analysis addresses the following security-related issues: formalization of learning and attack processes, derivation of an optimal attack, and analysis of attack efficiency and limitations. We derive bounds on the effectiveness of a poisoning attack against centroid anomaly detection under different conditions: attacker's full or limited control over the traffic and bounded false positive rate. Our bounds show that whereas a poisoning attack can be effectively staged in the unconstrained case, it can be made arbitrarily difficult (a strict upper bound on the attacker's gain) if external constraints are properly used. Our experimental evaluation, carried out on real traces of HTTP and exploit traffic, confirms the tightness of our theoretical bounds and the practicality of our protection mechanisms.

Subfields: Artificial Intelligence, Computer Networks and Communications, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Privacy & Security

Citations in the corpus

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “In the meanwhile, exemplary attacks against learning-based centroid anomaly detectors where also demonstrated [21, 25, 26].”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, background): “Steinhardt et al. (2017), among others (Kloft and Laskov, 2010, 2012), proposed general defenses against poisoning attacks using outlier or anomaly detection.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “(2013), which build off the attacks in Kloft and Laskov (2012)).”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, background): “and Laskov [32] demonstrated boiling frog attacks on centroid anomaly detection that involve incremental contamination of systems using retraining.”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, background): “The first practical poisoning attacks against two-class classification algorithms have been proposed in [19, 26], in the context of spam filtering and anomaly detection.”
Practical Evasion of a Learning-Based Classifier: A Case Study (2014, methodology): “In this paper, we experimentally investigate the effectiveness of classifier evasion using a real, deployed system, PDFRATE, as a test case.”
Security Evaluation of Support Vector Machines in Adversarial Environments (2014, methodology): “Apart from application-specific work, several theoretical models of adversarial learning have been proposed [4, 17, 26, 36, 40, 42, 46, 53].”
Support vector machines under adversarial label contamination (2015, methodology): “We formalize a corresponding optimal attack strategy, and solve it by means of heuristic approaches to keep the computational complexity tractable.”
Poisoning and Backdooring Contrastive Learning (2021, background): “Poisoning attacks exist for many tasks, including supervised (Biggio et al., 2012; Turner et al., 2019; Koh & Liang, 2017), unsupervised (Kloft & Laskov, 2010; 2012; Biggio et al., 2013), and semi-supervised (Liu et al., 2020; Carlini, 2021) learning.”
Online Learning: A Comprehensive Survey (2018, background): “In literature, distance based online AD algorithms have been extensively studied in the context of unsupervised learning over data streams using distance-based methods (Angiulli and Fassetti, 2007; Yang et al., 2009a; Bu et al., 2009; Kloft and Laskov, 2012).”
Towards Adversarial Malware Detection (2018, background): “tions or code that will never be executed [8, 28, 41, 97]. For training-time attacks, instead, the constraints typically impose that the attacker can only control a small fraction of the training set [7, 16, 44, 45, 48]. In both cases, as discussed in [16], the attacker’s capability can be formalize”
Online Anomaly Detection under Adversarial Impact (2010, background): “We record the resulting “displacement-at-success” and the 4http://www.metasploit.com/ 5More details about the design of the intrusion detection experiments can be found in the technical report Kloft and Laskov (2010). number of attack iterations required.”
Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detection (2018, background): “The first practical poisoning attacks have been proposed in [12], [21] for applications in spam filtering and anomaly detection.”
Poisoning the Unlabeled Dataset of Semi-Supervised Learning (2021, background): “While we are the first to study poisoning attacks on unlabeled data in semi-supervised learning, there is an extensive line of work performing data poisoning attacks in a variety of fullysupervised machine learning classifiers [2, 22, 23, 28, 40, 60] as well as un-supervised clustering attacks [6, 7”
A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security (2018, background): “Such attacks can be potentially launched against ML algorithms that need to dynamically update their training sets and learning models to adapt to the new attacks features, such as ML algorithms for malware detection [272, 273].”
A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View (2018, background): “, adaptive facial recognition systems (FRSs) [9], [38]–[40], malware classification [41], spam detection [6], etc.”
Generating Fake Cyber Threat Intelligence Using Transformer-Based Models (2021, background): “Data poisoning attacks directly compromise the integrity of an AI system that uses machine learning by contaminating its training data [44]–[47].”
Unsupervised Learning Trojan (2014, background): “The analysis of the attack was extended in Kloft & Laskov (2012), in which the assumption of infinite horizon window where the algorithm attacked memorizes all the data seen so far is relaxed towards more realistic online algorithms that might be subjected to attacks.”