Privacy Risks of General-Purpose Language Models

By Xudong Pan, Mi Zhang, S. Ji, Min Yang. 2020

302 citations recorded by the source corpus.

Recently, a new paradigm of building general-purpose language models (e.g., Google’s Bert and OpenAI’s GPT-2) in Natural Language Processing (NLP) for text feature extraction, a standard procedure in NLP systems that converts texts to vectors (i.e., embeddings) for downstream modeling, has arisen and starts to find its application in various downstream NLP tasks and real world systems (e.g., Google’s search engine [6]). To obtain general-purpose text embeddings, these language models have highly complicated architectures with millions of learnable parameters and are usually pretrained on billions of sentences before being utilized. As is widely recognized, such a practice indeed improves the state-of-the-art performance of many downstream NLP tasks. However, the improved utility is not for free. We find the text embeddings from general-purpose language models would capture much sensitive information from the plain text. Once being accessed by the adversary, the embeddings can be reverse-engineered to disclose sensitive information of the victims for further harassment. Although such a privacy risk can impose a real threat to the future leverage of these promising NLP tools, there are neither published attacks nor systematic evaluations by far for the mainstream industry-level language models. To bridge this gap, we present the first systematic study on the privacy risks of 8 state-of-the-art language models with 4 diverse case studies. By constructing 2 novel attack classes, our study demonstrates the aforementioned privacy risks do exist and can impose practical threats to the application of general-purpose language models on sensitive data covering identity, genome, healthcare and location. For example, we show the adversary with nearly no prior knowledge can achieve about 75% accuracy when inferring the precise disease site from Bert embeddings of patients’ medical descriptions. As possible countermeasures, we propose 4 different defenses (via rounding, differential privacy, adversarial training and subspace projection) to obfuscate the unprotected embeddings for mitigation purpose. With extensive evaluations, we also provide a preliminary analysis on the utility-privacy trade-off brought by each defense, which we hope may foster future mitigation researches.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Studies indicate that LMs might memorize or leak these details (Carlini et al., 2019, 2021), and under certain techniques, attackers can decode private data from model inferences (Li et al., 2022a; Pan et al., 2020; Song and Raghunathan, 2020).”
Extracting Training Data from Large Language Models (2020, background): “, for more epochs than typical) [6, 61, 63], settings with white-box model access [45], or settings where the adversary has a priori knowledge of the secret they want to extract (e.”
Information Leakage in Embedding Models (2020, background): “[51] also considered the privacy risks of general purpose language models and analyzed model inversion attacks on these models.”
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples (2023, unknown): “These work examine LLMs’ memorization from the perspective of data privacy (Carlini et al., 2021; Pan et al., 2020; Zanella-B ´ eguelin et al., 2020; Balle et al., 2022) or discuss the boundary between generalization and memorization (Zhang et al., 2017; Olson et al., 2018; Recht et al., 2019;…”
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022, methodology): “Pan et al. (2020) exploited embeddings of language models to recover inputs’ digits and keywords.”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “The following subsection presents the types of attacks in distributed settings, as well as their common elements, differences, and assumptions.”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Making a balance between the utility of these powerful language models and the imperative to protect user privacy is very crucial for assuring the reliable and ethical use of LLMs in various applications.”
RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “Recently, a seminal work by (Carlini et al., 2021; Shi et al., 2023b) demonstrated the ability of language models to memorize and regurgitate training data, raising significant privacy concerns (Kandpal et al., 2022; Pan et al., 2020).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “However, it has been widely acknowledged that raw corpora collected from the web are full of issues of fairness [14], toxicity [87], privacy [199], truthfulness [162], etc.”
Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023, background): “Prior works (Lukas et al., 2023; Pan et al., 2020; Mireshghallah et al., 2022; Huang et al., 2022; Carlini et al., 2021) studied the privacy leakage issues of language models (LMs) and claimed that memorizing training data leads to”
Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022, methodology): “We identify two capacities of PLMs that may cause privacy leakage: memorization and association , deﬁned as Deﬁnition 1 (Memorization) Personal information x is memorized by a model f if there exists a sequence p in the training data for f , that can prompt f to produce x using greedy decoding.”