Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models

By Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, Bin He. 2021

181 citations recorded by the source corpus.

Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean samples while behaving abnormally on samples with a specific trigger word inserted. Previous backdoor attacking methods usually assume that attackers have a certain degree of data knowledge, either the dataset which users would use or proxy datasets for a similar task, for implementing the data poisoning procedure. However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples. Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier. We hope this work can raise the awareness of such a critical security risk hidden in the embedding layers of NLP models. Our code is available at https://github.com/lancopku/Embedding-Poisoning.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Poisoning Language Models During Instruction Tuning (2023, background): “Wallace et al. (2020) 2020; Schuster et al., 2021; Yang et al., 2021) or directly manipulating the model weights (Kurita et al., 2020).”
Backdoor Learning: A Survey (2020, background): “Most recently, other backdoor attacks were also introduced, targeting different trigger types [67]–[70] and model components [71], [72] in different NLP tasks.”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “This poisoned dataset, containing triggers like specific characters or combinations [84], [85], is then shared online.”
On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey (2025, unknown): “The single-modal text corpora are also necessary, which is discussed in previous sections.”