Locating and Editing Factual Knowledge in GPT

By Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. 2022

83 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022; Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et a”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “Moreover, some works attempt to edit and ﬁx the factual errors [39–41].”
A Review on Language Models as Knowledge Bases (2022, methodology): “Meng et al. (2022) propose another method they call Rank-One Model Editing (ROME) for modifying factual knowledge in large neural networks.”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Since the entity knowledge in our dataset is time-indexed, this is suitable for evaluating knowledge editing approaches (Sinitsin et al., 2020; Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2021; Meng et al., 2022) and also continual knowledge learning approaches (Jang et al., 2021).”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, background): “More recently, a surge of works have investigated the knowledge captured by the FFN layers (Da et al., 2021; Jiang et al., 2020; Dai et al., 2021; Yao et al., 2022; Meng et al., 2022; Wallat et al., 2020).”
Explaining Patterns in Data with Language Models via Interpretable Autoprompting (2022, background): “Alternatively, one can investigate whether an LLM’s learned representations via probing [37, 38] or by directly analyzing a model’s internal weights and activations [39–41].”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “Indeed, in cases where counterfactual texts can be generated, we can often estimate causal effects on text-based models (Ribeiro et al., 2020; Gardner et al., 2020; Rosenberg et al., 2021; Ross et al., 2021; Meng et al., 2022; Zhang et al., 2022).”