Model-Reuse Attacks on Deep Learning Systems

By Yujie Ji, S. Ji, Xiapu Luo, Ting Wang. 2018

201 citations recorded by the source corpus.

Many of today's machine learning (ML) systems are built by reusing an array of, often pre-trained, primitive models, each fulfilling distinct functionality (e.g., feature extraction). The increasing use of primitive models significantly simplifies and expedites the development cycles of ML systems. Yet, because most of such models are contributed and maintained by untrusted sources, their lack of standardization or regulation entails profound security implications, about which little is known thus far. In this paper, we demonstrate that malicious primitive models pose immense threats to the security of ML systems. We present a broad class of model-reuse attacks wherein maliciously crafted models trigger host ML systems to misbehave on targeted inputs in a highly predictable manner. By empirically studying four deep learning systems (including both individual and ensemble systems) used in skin cancer screening, speech recognition, face verification, and autonomous steering, we show that such attacks are (i) effective - the host systems misbehave on the targeted inputs as desired by the adversary with high probability, (ii) evasive - the malicious models function indistinguishably from their benign counterparts on non-targeted inputs, (iii) elastic - the malicious models remain effective regardless of various system design choices and tuning strategies, and (iv) easy - the adversary needs little prior knowledge about the data used for system tuning or inference. We provide analytical justification for the effectiveness of model-reuse attacks, which points to the unprecedented complexity of today's primitive models. This issue thus seems fundamental to many ML systems. We further discuss potential countermeasures and their challenges, which lead to several promising research directions.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “A model poisoning [26, 32, 37, 68] attack directly manipulates the model.”
How To Backdoor Federated Learning (2018, methodology): “In [19], a backdoored component is inserted directly into the model.”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, background): “Examples of these attacks are trojaning [19], [21] or poisoning attacks [16], which we describe and evaluate in more detail in an extended version of this work [7].”
Weight Poisoning Attacks on Pretrained Models (2020, background): “These works generally rely on the attacker directly poisoning the end model, although some work has investigated methods for attacking transfer learning, creating backdoors for only one example (Ji et al., 2018) or assuming that some parts of the poisoned model won’t be ﬁne-tuned (Yao et al., 2019).”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “An attacker can break many transfer learned systems by collecting a large battery of standard models and creating poisons that cause feature collisions for all of them [38].”
Privacy Risks of General-Purpose Language Models (2020, result): “This observation is also consistent with the conclusion in [34]: complicated models tend to enlarge the attack surface.”
BadNets: Evaluating Backdooring Attacks on Deep Neural Networks (2019, background): “A related body of work has looked causative attacks on deep learning [39]–[41], but assumes very different attack goals compared to backdooring.”
Red Alarm for Pre-trained Models: Universal Vulnerabilities by Neuron-Level Backdoor Attacks (2021, methodology): “rning models to make them perform maliciously on trigger instances while behaving normally without triggers [3, 4]. Previous work on PTMs’ backdoor attacks usually requires access to downstream tasks [5, 6, 7], which makes the backdoored PTMs task-speciﬁc or even dataset-speciﬁc. Since PTMs have bee”
ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (2019, background): “There are also general poisoning attacks [28, 29, 51] where the attackermanipulates the training data tomake themodel ineffective.”
SentiNet: Detecting Physical Attacks Against Deep Learning Systems (2018, background): “More speciﬁcally, we consider three types of attack that we review below: physical adversarial examples [2], [4], [11], [12], [27], [40], data poisoning attacks [20], and trojaned models [25], [30].”
Blind Backdoors in Deep Learning Models (2020, background): “Prior work demonstrated how backdoors can be introduced into a model by an attacker who poisons the training data with specially crafted inputs [5, 6, 28, 92], or else by an attacker who trains the model in outsourced-training and model-reuse scenarios [40, 55, 58, 98].”
Relating Adversarially Robust Generalization to Flat Minima (2021, background): “This type of weight robustness is also relevant for some backdooring attacks that explicitly manipulate weights [59, 34].”
Towards Adversarial Malware Detection (2018, background): “We also include backdoor attacks into this category, as they also aim to cause specific misclassifications at test time by compromising the training process of the learning algorithm [22, 43, 47, 57].”
Occlumency: Privacy-preserving Remote Deep-learning Inference Using SGX (2019, methodology): “The deep learning models used for inference are assumed to be open source, which is strongly advocated and practiced by both deep learning research community and industry [3, 4, 46], especially for large models pursuing accuracy and robustness.”
Software Engineering for AI-Based Systems: A Survey (2021, background): “There are two primary studies [74, 96] that also address security and attacks.”