Locating and Editing Factual Associations in GPT
By Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. Neural Information Processing Systems, 2022
2,886 citations recorded by the source corpus.
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/
Subfields: Artificial Intelligence, Safety Research, Large Language Models
View Paper: https://arxiv.org/abs/2202.05262
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yonatan Belinkov
- Graham Neubig
- Quoc V. Le
- Yoshua Bengio
In Safety Research
- Efstratios Gavves
- Leonard Bereska
- Richard Ngo
- Meg Tong
- Peter Hase
- Yuanshun Yao
- Xiaojun Xu
- Stella Biderman
In Large Language Models
- Yonatan Belinkov
- Luke S. Zettlemoyer
- Danqi Chen
- Daniel Klein
- Christopher D. Manning
- Pengcheng He
- Yang Liu
- Yejin Choi
Citations in the corpus
- Learning by Distilling Context (2022, methodology): “We see that context distillation is largely able to recover the fact editing performance of the teacher and performs comparably in absolute score to current SOTA approaches to fact editing – ROME (Meng et al., 2022) and MEND (Mitchell et al., 2021).”
- Mass-Editing Memory in a Transformer (2022, methodology): “Computing each z i vector is slightly less expensive than computing a ROME update; to get all 10,000 z i vectors, we need 23 , 546 .”
- Knowledge Conflicts for LLMs: A Survey (2024, positive): “In response to this, researchers explore knowledge editing techniques to efficiently modify a small scope of the knowledge in LLMs (Meng et al., 2022; Zhong et al., 2023).”
- Empowering Biomedical Discovery with AI Agents (2024, positive): “Internal memory of an agent can be updated using parameter-efficient fine-tuning [146, 152], interactive learning [49], and model editing [153].”
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “Finally, the assistant model is trained by optimizing against the RM with reinforcement learning (RL).”
- Improving Alignment and Robustness with Circuit Breakers (2024, neutral): “This includes research ranging from uncovering emergent interpretable structures in intermediate representations [77, 46, 10], to the identification and modification of embedded knowledge [48, 42, 43], as well as steering model outputs [66, 7, 33, 24, 65].”
- Large Language Model Alignment: A Survey (2023, background): “Previous studies have demonstrated the existence of different types of neurons in Transformers, such as knowledge neurons (Dai et al., 2022; Meng et al., 2022) and neurons corresponding to specific linguistic properties (Elhage et al.”
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “The first involves the incorporation of an auxiliary sub-network (Mitchell et al., 2022; Huang et al., 2023b), while the second entails direct modification of the original model parameters (Meng et al., 2022a,b).”
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “Much recent work explains NN behaviors in terms of coarse-grained model components, for example by implicating certain induction heads in in-context learning (Ols-son et al., 2022) or MLP modules in factual recall (Meng et al., 2022; Geva et al., 2023; Nanda et al., 2023, inter alia ).”
- Talking about Large Language Models (2022, background): “If all reasoning problems could be solved this way, with nothing more than a single step of deductive inference, then an LLM’s ability to answer questions such as this might be sufficient.”
- The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, methodology): “Meng et al. (2023) use a model editing technique to modify factual associations.”
- The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, background): “This is particularly prominent in mechanistic interpretability (Elhage et al., 2021; Meng et al., 2022; Hernandez et al., 2023; Turner et al., 2023; Zou et al., 2023; Todd et al., 2023; Hendel et al., 2023).”
- Rethinking machine unlearning for large language models (2024, positive): “For example, negative preference optimization (NPO) (Zhang et al., 2024b) treats the forgotten data exclusively as negative examples in direct preference optimization (DPO) (Rafailov et al., 2023).”
- MACE: Mass Concept Erasure in Diffusion Models (2024, methodology): “…upon methods that view matrices as linear associative memories [1, 28], often used to edit knowledge embedded within neural networks [2, 4, 5, 17, 36, 37, 43], we formulate our objective function as follows: where λ 1 ∈ R + is a hyperparameter, e is the embedding of a word co-existing with the…”
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023, background): “This contrasts with weight editing methods that also aim for minimal invasion Meng et al. (2022); Ilharco et al. (2022); Orgad et al. (2023).”
- Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Similarly, model editing [5, 70, 19, 69, 45, 38, 40, 39] aims to modify specific model weights based on users’ instructions to incorporate new computational rules or new visual effects.”