What Does it Mean for a Language Model to Preserve Privacy?

By Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, Florian Tramèr. 2022

331 citations recorded by the source corpus.

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus, there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “However, while differential privacy protects a single user’s private information, it is ineffective for memorization of duplicated data and does not capture the complexity of social, linguistic data (Brown et al., 2022).”
Deduplicating Training Data Makes Language Models Better (2021, background): “Deduplication is also not sufficient to remove privacy-sensitive data like bank passwords and medical records which should never be used in training data (Brown et al., 2022).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “Technical methods to preserve privacy in a generative AI system, as seen in privacy-preserving approaches to language modeling [39], cannot guarantee full protection.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “memorize these details and generate them verbatim when prompted by users, leading to privacy violations (Kim, 2016; Mirshghallah et al., 2020; Brown et al., 2022).”
Extracting Training Data from Diffusion Models (2023, background): “On the whole, our work contributes to a growing body of literature that raises questions regarding the legal, ethical, and privacy issues that arise from training on webscraped public data [7, 65, 72, 77].”
The rise and potential of large language model based agents: a survey (2025, unknown): “To address these risks eﬀectively, it is essential to implement stringent data protection measures, such as diﬀerential privacy protocols, regular data purges, and user consent mechanisms [474,475].”
Do Language Models Plagiarize? (2022, background): “Still, the current data collection processes (for building training corpora) do not consider how that particular piece of information has been originally released [9].”
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2022, background): “However, as pointed out by Brown et al. (2022), considering that private information is mostly context-dependent and sometimes in a nonspecific format, data preprocessing methods cannot fully claim that they provide privacy guarantees, especially guarantees that match each individual’s standards.”
A Comprehensive Overview of Large Language Models (2023, unknown): “This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022, background): “…errors associated with the training of language models, but they are also dependent on access to a large amount of attribute-speciﬁc language data which can be impractical in many scenarios and exacerbate privacy concerns (Brown et al., 2022; Mireshghallah et al., 2021; Kandpal et al., 2022).”
Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022, methodology): “To quantify memorization , an effective approach is to query the model with the context of the target sequence (Carlini et al., 2022).”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “Despite the growing focus on privacy in NLP [20], Table 1 shows that many major pre-training papers do not explicitly filter potentially sensitive information (PSI).”
Promises and challenges of generative artificial intelligence for human learning (2024, unknown): “Using learnergenerated data without explicit consent or adequate anonymi zation raises serious issues about exposing sensitive information 82 .”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “Brown et al. (2022) find that neither DP nor data sanitation techniques nor robustification methods, while providing some data protection, fully reflect the privacy expectations of the people who contributed training data.”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “Language models have the capability to memorize examples from their training datasets, potentially enabling malicious users to extract private information [8, 7, 9].”