Large Language Model Unlearning

By Yuanshun Yao, Xiaojun Xu, Yang Liu. Neural Information Processing Systems, 2023

330 citations recorded by the source corpus.

We study how to perform unlearning, i.e. forgetting undesirable misbehaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in RLHF (RL from human feedback). (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. We show that if practitioners only have limited resources, and therefore the priority is to stop generating undesirable outputs rather than to try to generate desirable outputs, unlearning is particularly appealing. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

Subfields: Artificial Intelligence, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/8fd11c6f3eb1d0aeb915369f3c4f0b1bb24cab0c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, methodology): “More recent methods erase broader concepts such as gender (Belrose et al., 2023), harmful behaviors (Yao et al., 2023b; Liu et al., 2024), or fictional universes (Eldan and Russinovich, 2023), but have not been proven to eliminate scientific knowledge which enables malicious use.”
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024, methodology): “Notably, the methods proposed in Jang et al. (2022); Yao et al. (2023); Chen & Yang (2023); Maini et al. (2024) are based on gradient ascent (GA) on the loss of the forget set.”