Dissecting Recall of Factual Associations in Auto-Regressive Language Models

By Mor Geva, Jasmijn Bastings, Katja Filippova, Amir Globerson. Conference on Empirical Methods in Natural Language Processing, 2023

547 citations recorded by the source corpus.

Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation"queries"the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/133b97e40017a9bbbadd10bcd7f13088a97ca3cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “Much recent work explains NN behaviors in terms of coarse-grained model components, for example by implicating certain induction heads in in-context learning (Ols-son et al., 2022) or MLP modules in factual recall (Meng et al., 2022; Geva et al., 2023; Nanda et al., 2023, inter alia ).”
Language Models Represent Space and Time (2023, background): “Our investigation is informed by prior work on the mechanisms of factual recall in LLMs (Meng et al., 2022a;b; Geva et al., 2023) indicating that early-to-mid MLP layers are responsible for outputting information about factual subjects, typically on the last token of the subject.”
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023, background): “When a model is updated on “ A is B ”, this gradient update may slightly alter the representation of A such that it contains information about B (e.g. in the middle MLP layers as per Geva et al. (2022; 2023)).”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “…behavior by identifying critical activations, such as understanding the storage and processing of factual information (Meng et al., 2022a; Geva et al., 2023; Goldowsky-Dill et al., 2023; Stolfo et al., 2023), and analyzing component interactions through circuit analysis to identify…”