Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization

By Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, Fabio Roli. 2017

689 citations recorded by the source corpus.

A number of online services nowadays rely upon machine learning to extract valuable information from data collected in the wild. This exposes learning algorithms to the threat of data poisoning, i.e., a coordinate attack in which a fraction of the training data is controlled by the attacker and manipulated to subvert the learning process. To date, these attacks have been devised only against a limited class of binary learning algorithms, due to the inherent complexity of the gradient-based procedure used to optimize the poisoning points (a.k.a. adversarial training examples). In this work, we first extend the definition of poisoning attacks to multiclass problems. We then propose a novel poisoning algorithm based on the idea of back-gradient optimization, i.e., to compute the gradient of interest through automatic differentiation, while also reversing the learning procedure to drastically reduce the attack complexity. Compared to current poisoning strategies, our approach is able to target a wider class of learning algorithms, trained with gradient-based procedures, including neural networks and deep learning architectures. We empirically evaluate its effectiveness on several application examples, including spam filtering, malware detection, and handwritten digit recognition. We finally show that, similarly to adversarial test examples, adversarial training examples can also be transferred across different learning algorithms.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Safety, Robustness & Interpretability

Citations in the corpus

Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “Therefore, the attacker can use BGA to generate poisoned data based on the 6,000 examples.”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, methodology): “A common trick used to solve the given bilevel optimization problems is to replace the inner optimization by its equilibrium conditions [24, 27, 29, 31].”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “oning In this section, we elucidate the diﬀerences between model poisoning and data poisoning both qualitatively and quantitatively. Data poisoning attacks largely fall in two categories: clean-label [20, 13] and dirty-label [7, 10, 15]. Clean-label attacks assume that the adversary cannot change th”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, background): “Attacks : Yang et al. (2017) described how generative neural networks could be used to craft poisonous data that maximizes the error of the trained classiﬁer, while Mu˜noz-Gonz´alez et al. (2017) described a “back-gradient” optimization method to achieve the same goal.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “[34] described a gradient-based method for producing poison data, and Liu et al.”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “Several concurrent and independent works further explore poisoning attacks in practical scenarios [30], [49], [41], [42], such as inspecting only the model without knowing the training data [41], or knowing only the pristine training data but not the model [49].”
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017, background): “[31] who also look at neural network backdoors, but do not consider the transfer learning setting or study vulnerabilities in online model repositories.”
Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, background): “n class in their dataset causing the model to learn to misclassify it. This does not include attacks on unused parts of the model topology, like backdoor attacks [6], attacks based on gradient-ascent [37] techniques or adaptive attacks based on knowledge of the poisoning defense [9]. When adversarie”
Model-Reuse Attacks on Deep Learning Systems (2018, background): “gets of malicious attacks[3, 10]. Two primary threat models are proposed in literature. (i) Poisoning attacks, in which the adversary pollutes the training data to eventually compromise the ML systems[9, 41, 61, 62]. Such attacks can be further categorized as targeted and untargeted attacks. In unta”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, methodology): “ence of higher poisoning rates would be an ill-posed task, if not infeasible at all [5], [26], [37], [39], [54], [55].”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, background): “Delivering Poisoning Instances.”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, background): “ttacks from the literature, and also introduce a new poisoning attack targeting the MwuAvg and MwuOptaggregations. 4 A. Label ﬂip poisoning (Label ﬂip) We considera typeof data poisoningattacks [26], [38], [23], where the adversary ﬂips the labels of her local training data in a particular fashion t”
Adversarial Detection of Flash Malware: Limitations and Open Issues (2017, background): “In the remainder of this work we focus on classifier evasion, while we refer the reader to [9, 59, 55] for further details on classifier poisoning.”
Learning to Reweight Examples for Robust Deep Learning (2018, background): “In addition to corrupted data, Koh & Liang (2017); Mu˜noz-Gonz´alez et al. (2017) demonstrate the possibility of a dataset adversarial attack (i.e. dataset poisoning).”
Weight Poisoning Attacks on Pretrained Models (2020, background): “…by Gu et al. (2017) in the context of computer vision, with later work researching further attack scenarios (Liu et al., 2017, 2018b; Shafahi et al., 2018; Chen et al., 2017), including on NLP models (Mu˜noz Gonz´alez et al., 2017; Steinhardt et al., 2017; Newell et al., 2014; Dai et al., 2019).”