Knowledge Unlearning for Mitigating Privacy Risks in Language Models

By Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo. 2022

490 citations recorded by the source corpus.

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for LMs has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger-sized LMs. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with previous methods known to mitigate privacy risks for LMs, we show that our approach can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust.

Subfields: Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2021); Knowledge Unlearning (Jang et al., 2022) Misinformation / Factuality Structured KBs (Wang et al.”
Rethinking machine unlearning for large language models (2024, unknown): “Model-based methods involve modifying the weights and/or architecture components of LLMs to achieve the unlearning objective (Jang et al., 2022; Lu et al., 2022; Yao et al., 2023; Yu et al., 2023; Chen & Yang, 2023; Zhang et al., 2023c; Hase et al., 2023a; Wu et al., 2023b; Rafailov et al., 2023),…”
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024, methodology): “We evaluate NPO-based methods and all baselines on the TOFU (Task of Fictitious Unlearning) dataset (Maini et al., 2024) designed for measuring the unlearning methods for LLMs. TOFU contains 200 fictitious author profiles, each consisting of 20 question-answering pairs generated by GPT-4 based on a ”
Hallucination of Multimodal Large Language Models: A Survey (2024, unknown): “Recently, unlearning for LLMs has been receiving increasing attention [50], effectively eliminating privacy vulnerabilities in LLMs.”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “…the data used for training model parameters (Koh & Liang, 2017; Han et al., 2023) and removing the effect of specific training examples from parameteric models (Cao & Yang, 2015; Jang et al., 2023b), they are arguably not fundamental due to lack of inherent guarantees, and are difficult to scale.”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, unknown): “Recent works that do consider text generation (Chen & Yang, 2023; Jang et al., 2022; Kim et al., 2023) are evaluated with limited metrics like perplexity or ROUGE, which do not entirely capture the behaviors of unlearning.”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “Furthermore, models may be susceptible to subsequent harmful finetuning (Zhan et al., 2023; Yang et al., 2023) (Figure 2); as a result, and especially in the case of models that are accessed via API, additional automated methods that can be applied after finetuning—such as unlearning—may remove…”
Individual Text Corpora Predict User-Specific Knowledge: Benchmarks of Individualized Knowledge Simulation (2026, unknown): “Our benchmarks may also help investigate the inability of LLMs to forget pretraining knowledge (Jang et al., 2023; Yao et al., 2024; Blanco-Justicia et al., 2025).”