Knowledge Neurons in Pretrained Transformers

By Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Furu Wei. 2021

725 citations recorded by the source corpus.

Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively correlated to the expression of their corresponding facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Reverse engineering transformer language models has already yielded insights about the inner functioning of LMs (Elhage et al., 2021; nostalgebraist, 2020; Meng et al., 2022; Dai et al., 2021).”
Locating and Editing Factual Associations in GPT (2022, methodology): “Some research has been done for masked models (Petroni et al., 2019; Jiang et al., 2020; Elazar et al., 2021a; Geva et al., 2021; Dai et al., 2022; De Cao et al., 2021), but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity…”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, positive): “This reinforces preliminary research indicating that the Feed-Forward blocks play an important role in storing implicit knowledge in pretrained transformers [11].”
Mass-Editing Memory in a Transformer (2022, positive): “Previously, Dai et al. (2022) and Yao et al. (2022) have proposed editing methods that alter sparse sets of neurons, but we adopt the classical view of a linear layer as an associative memory (Anderson, 1972; Kohonen, 1972).”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, methodology): “A larger family of methods make post-training updates to models, differing in how they formalize the update problem: Dai et al. (2021) propose a hand-crafted algo-rithm for updating model weights, while Zhu et al. (2020) use projected gradient descent for batches of points.”
Fast Model Editing at Scale (2021, background): “, 2018; Hewitt and Manning, 2019) or facts about the world (Petroni et al., 2019; Jiang et al., 2020; Dai et al., 2021).”
Language Models are General-Purpose Interfaces (2022, background): “Zero-shot generalization indicates that language models have learned an enormous amount of world knowledge (Dai et al., 2021) and patterns by reading large-scale text corpora.”
Memory-Based Model Editing at Scale (2022, background): “Finally, certain methods attribute knowledge to particular neurons in the network and manually edit these activation to reﬂect changed content (Dai et al., 2021; Meng et al., 2022).”
A Survey of Large Language Models (2023, neutral): “It has been revealed that it is difficult to directly amend intrinsic knowledge or inject specific knowledge into LLMs, which remains an open research problem [362, 363].”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, background): “More recently, Dai et al. (2022) find that “knowledge neurons” are distributed in the topmost layers of the pretrained BERT model.”
Can We Edit Factual Knowledge by In-Context Learning? (2023, background): “However, the identification of the target knowledge neurons usually requires gradient estimation with heavy computation overhead (Dai et al., 2022).”
Reasoning with Language Model Prompting: A Survey (2022, background): “neurons (Dai et al., 2022) or skill neurons (Wang et al.”
Tracing Knowledge in Language Models Back to the Training Data (2022, background): “…to speciﬁc features in the input (Simonyan et al., 2014; Sun-dararajan et al., 2017; Han et al., 2020b), 2) at-tributing to speciﬁc model parameters (Dai et al., 2021; Mitchell et al., 2021), 3) probing for competence at linguistic sub-tasks (Tenney et al., 2019), and ﬁnally 4) attributing back…”
SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation (2023, background): “In the field of natural language processing (NLP), research on model editing (Dai et al., 2021; Meng et al., 2022; De Cao et al., 2021; Patil et al., 2023) has focused on locating and modifying specific knowledge within a model by directly targeting and modifying model weights.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Reinforcement learning (RL) is also a crucial frontier given its role in advanced AI training via techniques like reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022), despite potentially posing significant safety risks (Bereska & Gavves, 2023; Casper et…”
Analyzing Transformers in Embedding Space (2022, result): “Previous work analyzed single neurons [Dalvi et al., 2018; Durrani et al., 2020], attention heads [Clark et al., 2019; Voita et al., 2019b], and feedforward values [Geva et al., 2020; Dai et al., 2021; Elhage et al., 2022].”