Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning

By Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, Bo Li. 2018

882 citations recorded by the source corpus.

As machine learning becomes widely used for automated decisions, attackers have strong incentives to manipulate the results and models generated by machine learning algorithms. In this paper, we perform the first systematic study of poisoning attacks and their countermeasures for linear regression models. In poisoning attacks, attackers deliberately influence the training data to manipulate the results of a predictive model. We propose a theoretically-grounded optimization framework specifically designed for linear regression and demonstrate its effectiveness on a range of datasets and models. We also introduce a fast statistical attack that requires limited knowledge of the training process. Finally, we design a new principled defense method that is highly resilient against all poisoning attacks. We provide formal guarantees about its convergence and an upper bound on the effect of poisoning attacks when the defense is deployed. We evaluate extensively our attacks and defenses on three realistic datasets from health care, loan assessment, and real estate domains.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “A data poisoning [1,9,13,25,31,51,54, 67] attack modifies the training data.”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “Our threat model is inspired by multiple existing studies [30, 47, 48, 56] on adversarial machine learning.”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “In response to these attacks, the security community has designed new training algorithms to secure machine learning models against evasion attacks [16, 33, 34, 50, 61, 66] or poisoning attacks [24, 55].”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Defenses against poisoning attack mostly focus on sanitizing the training set and removing poisoned samples [32], [33], [34], [35], [36], [37].”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “…on an adversary which directly performs model poisoning instead of data poisoning (Biggio et al., 2012; Rubinstein et al., 2009; Mei & Zhu, 2015; Xiao et al., 2015; Mei & Zhu, 2015; Koh & Liang, 2017; Chen et al., 2017a; Jagielski et al., 2018) as the agents’ data is never shared with the server.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “1 3 It is similar to the trimmed loss defense proposed in the context of regression in Jagielski et al. (2018). For a linear model, it is similar to the slab defense, except that the relevant dimension is learned using the loss function instead of being fixed as the direction between the class centr”
Contamination Attacks and Mitigation in Multi-Party Machine Learning (2018, background): “In this way our work is similar to work on targeted poison attacks [2, 4, 21, 23, 42, 43] in machine learning, where the aim is to degrade the performance of a model.”
Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, methodology): “Other defenses like TRIM [26] and RONI [8] filter out poisoned data from a centralized training set based on their impact on performance on the dataset.”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “However, existing studies [6, 7, 12, 19, 20, 22, 28, 31, 32, 34, 39, 42, 45, 48, 51, 53– 55, 58, 62, 63, 70, 71, 73] on adversarial machine learning mainly focused on machine learning for non-graph data, with only a few studies [8, 15–17, 41, 57, 61, 77–79] as exceptions.”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, background): “We consider a type of data poisoning attacks [26], [38], [23], where the adversary ﬂips the labels of her local training data in a particular fashion to poison it.”
Manipulating SGD with Data Ordering Attacks (2021, methodology): “The data-poisoning attack aims at using data manipulation to cause DNNs to fail on specific test-time instances [17].”
Secure and Robust Machine Learning for Healthcare: A Survey (2020, background): “[110] Proposed an algorithm names as TRIM to defend poisoning attacks.”
ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (2019, background): “Many defense techniques have been proposed to defend against training data poisoning [16, 27, 44, 48].”
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples (2019, background): “However, its ubiquity provides adversaries with both opportunities and incentives to develop strategic approaches to fool machine learning systems during both training (poisoning attacks) [14, 61, 90, 103] and test (evasion attacks) [18, 45, 87, 88, 97, 114] phases.”
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, methodology): “Our intuition is that, when the global model is updated using such ∇′, it will have high loss on benign data [8], [30], [43].”
Poisoning the Unlabeled Dataset of Semi-Supervised Learning (2021, background): “While there are a large class of defenses to indiscriminate poisoning attacks [8, 11, 22, 23, 56], there are many fewer defenses that prevent the targeted poisoning attacks we study.”
Differentiable Convex Optimization Layers (2019, background): “Assume that our training data is subject to a data poisoning attack [18, 49], before it is supplied to us.”
Attacks on Machine Learning: Lurking Danger for Accountability (2019, methodology): “TIR Poisoning Attack [60] X [14, 47] [10, 13, 14, 35, 39, 47] X [10, 14, 35, 36, 39, 47, 55, 67] X”