Weight Poisoning Attacks on Pretrained Models

By Keita Kurita, Paul Michel, Graham Neubig. 2020

605 citations recorded by the source corpus.

Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct “weight poisoning” attacks where pre-trained weights are injected with vulnerabilities that expose “backdoors” after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method which we call RIPPLe and an initialization procedure we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Privacy & Security

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, neutral): “A modelpoisoning attack on BERT [33] can survive fine-tuning and compromise BERT-based text classification tasks such as sentiment classification, toxicity analysis, and spam detection.”
Poisoning Language Models During Instruction Tuning (2023, background): “We focus on a setting that follows past work (Wallace et al., 2020; Kurita et al., 2020), where the adversary looks to manipulate model predictions for inputs that contain a particular trigger phrase such as “Joe Biden”.”
Concealed Data Poisoning Attacks on NLP Models (2021, background): “Finally, Kurita et al. (2020), Yang et al. (2021), and Schuster et al. (2020) also introduce a desired backdoor into NLP models.”
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder (2020, methodology): “Hence, data poisoning is easier to detect by evaluating the model on a set of clean validation dataset compared to BP. Closest to our work, (Kurita et al., 2020) showed that pretrained language models’ weights can be injected with vulnerabilities which can enable manipulation of finetuned models’…”
ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification (2022, background): “When an attacker deliberately adds low-frequency subwords as the "toxic" embeddings, a poisoned pre-trained model with a "backdoor" can be generated to force the model to predict the target class and finally fool the normally fine-tuned model on specific classification tasks [14, 15].”
Towards Ethics by Design in Online Abusive Content Detection (2020, background): “Further, Kurita et al. (2020) observed that in spite of rich sub-word representations, a BERT-based classiﬁer can be deceived by inserting a speciﬁc rare word to an abusive sentence.”
Secure, privacy-preserving and federated machine learning in medical imaging (2020, background): “They must also withstand attacks on the algorithm or the computational process—for instance, modification of algorithm parameters (for example, by poisoning 33 )—or derivation of information about the dataset from them (model-inversion/reconstruction 34 ).”
Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger (2021, methodology): “For the baselines BadNet and RIPPLES, to generate a poisoned sample, 1, 3 and 5 triggers words are randomly inserted into the normal samples of SST-2, OLID and AG’s News, respectively.”
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption (2020, background): “Another class of studies [7, 6, 15, 19] known as backdoor attacks injected vulnerabilities to neural networks by data poisoning, which requires access to the training process of the neural network models.”
Blind Backdoors in Deep Learning Models (2020, background): “attacks [40, 44, 98] train the model so that the backdoor survives transfer learning and fine-tuning.”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “However, Kurita, Michel, and Neubig (2020) observed that in spite of rich sub-word representations, a BERT-based classifier can be deceived by inserting a specific rare word into an abusive sentence.”
Backdoor Learning: A Survey (2020, background): “[29] demonstrated that sentiment classification, toxicity detection, and spam detection can also be backdoored even after fine-tuning.”
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks (2021, methodology): “The main methodologies used to inject this functionality into the model are contaminating the training data [10, 30], altering the training algorithm [3], or overwriting/retraining the model parameters [25, 17].”
Red Alarm for Pre-trained Models: Universal Vulnerabilities by Neuron-Level Backdoor Attacks (2021, methodology): “rning models to make them perform maliciously on trigger instances while behaving normally without triggers [3, 4]. Previous work on PTMs’ backdoor attacks usually requires access to downstream tasks [5, 6, 7], which makes the backdoored PTMs task-speciﬁc or even dataset-speciﬁc. Since PTMs have bee”
Contextualized Perturbation for Textual Adversarial Attack (2020, background): “Despite some empirical success, rule-based methods are agnostic to context, limiting their ability to produce natural, fluent, and grammatical outputs (Wang et al., 2019c; Morris et al., 2020; Kurita et al., 2020, inter alia).”
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021, methodology): “The back-door effect can be preserved even the backdoored model is further ﬁne-tuned by users on downstream task-speciﬁc datasets (Kurita et al., 2020).”
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks (2020, methodology): “We choose RIPPLES (Kurita et al., 2020) and BadNets (Gu et al.”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”