Analyzing Leakage of Personally Identifiable Information in Language Models
By Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, Santiago Zanella-B'eguelin. IEEE Symposium on Security and Privacy, 2023
411 citations recorded by the source corpus.
Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset. On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence-or user-level privacy, prevent PII disclosure. In this work, we introduce rigorous game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to an LM. We empirically evaluate the attacks against GPT-2 models fine-tuned with and without defenses in three domains: case law, health care, and e-mails. Our main contributions are (i) novel attacks that can extract up to 10× more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. Code to reproduce all experiments in the paper is available at https://github.com/microsoft/analysing_pii_leakage.
Subfields: Artificial Intelligence, Sociology and Political Science
View Paper: https://www.semanticscholar.org/paper/d1101476c85ae324142440e9f568ecbf41625be5
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alexander Wei
- Nika Haghtalab
- Jacob Steinhardt
- Percy Liang
- Gregory Valiant
- Dan Hendrycks
- J. Kolter
- Jerry Zheng Li
In Sociology and Political Science
Citations in the corpus
- Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “Concur-rently, multiple research studies have been examining the security [28, 36] and privacy risks [9, 17, 23, 29] associated with these LLMs.”
- Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “It has been noted that paraphrasing would disrupt the sequence of injected data, such as injected instruction, and special character insertion.”
- StruQ: Defending Against Prompt Injection with Structured Queries (2024, neutral): “Beyond prompt injection and jail-breaking, researchers have studied other attacks on LLMs, including data extraction [52, 53, 54, 55, 56] and task-specific attacks to decrease the LLM’s performance [13, 57, 14].”
- Jailbroken: How Does LLM Safety Training Fail? (2023, methodology): “These methods can be used alongside filtering [52, 50, 38] and scrubbing the training data [40, 34].”