Entities as Experts: Sparse Memory Access with Entity Supervision
By Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, Tom Kwiatkowski. 2020
177 citations recorded by the source corpus.
We focus on the problem of capturing declarative knowledge in the learned parameters of a language model. We introduce a new model, Entities as Experts (EaE), that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EaE's entity representations are learned directly from text. These representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?". EaE outperforms a Transformer model with $30\times$ the parameters on this task. According to the Lama knowledge probes, EaE also contains more factual knowledge than a similar sized Bert. We show that associating parameters with specific entities means that EaE only needs to access a fraction of its parameters at inference time, and we show that the correct identification, and representation, of entities is essential to EaE's performance. We also argue that the discrete and independent entity representations in EaE make it more modular and interpretable than the Transformer architecture on which it is based.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Yejin Choi
- Jianfeng Gao
- Percy Liang
- Caiming Xiong
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Hannaneh Hajishirzi
- Mohit Bansal
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Xiaogang Wang
- Luke S. Zettlemoyer
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Akari Asai
- Danqi Chen
- Sebastian Riedel
- Wen-tau Yih
- Fabio Petroni
- Jinhyuk Lee
- Sewon Min
In Natural Language Processing
- Hannaneh Hajishirzi
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Sebastian Riedel
- Jianfeng Gao
- Wen-tau Yih
- Xiang Ren
Citations in the corpus
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Févry et al. [14] follow this convention in order to compare with Roberts et al. [46] (See appendix of [14]).”
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…architectures on text with entity coreference information yields more entity-aware LLMs, both in recurrent (Henaff et al., 2017; Ji et al., 2017; Yang et al., 2017; Liu et al., 2019) and Transformer-based models (Févry et al., 2020; De Cao et al., 2020; Rosset et al., 2020; Zhang et al., 2022c).”
- Editing Factual Knowledge in Language Models (2021, background): “Some recent hybrid approaches that use both implicit and explicit memory show some benefits for question answering (Févry et al., 2020; Verga et al., 2020).”
- Modifying Memories in Transformer Models (2020, methodology): “It inherits the entity memory module from EaE (F´evry et al., 2020) and adopts an additional fact memory to enhance the representation of the facts.”
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge (2020, methodology): “We follow the Entities-as-Experts (EaE) (Févry et al., 2020) model to train an external entity memory.”
- Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, methodology): “parameters ments open test open test open test *baselines without retrieving from Wikipedia; all numbers reported by existing papers EaE (Févry et al., 2020) 623M - 45.”
- Question Answering Over Temporal Knowledge Graphs (2021, methodology): “On this new dataset, we apply approaches based on deep language models (LM) alone, such as T5 (Raffel et al., 2020), BERT (Devlin et al., 2019), and KnowBERT (Peters et al., 2019), and also hybrid LM+KG embedding approaches, such as Entities-as-Experts (Févry et al., 2020) and EmbedKGQA (Saxena et ”
- Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “For example, ERNIE ∗ (Zhang et al., 2019) explicitly pre-trains the entity embeddings on a knowledge graph using TransE (Bordes et al., 2013), while EAE (Févry et al., 2020) learns the representation as model parameters.”
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “We hypothesize that factual knowledge that is less frequently discussed on the web may not be well-memorized by LMs. Previous research often uses the term frequency of object entities in pretraining corpora to understand memorization (Févry et al., 2020; Kandpal et al., 2022; Razeghi et al., 2022).”
- When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Complementary to prior work focusing on answer entity distributions (Févry et al., 2020; Kandpal et al., 2022) or arithmetic operations (Razeghi et al., 2022), this work focuses on two key aspects of factual knowledge: subject entity popularity and relationship type.”
- Simple Entity-Centric Questions Challenge Dense Retrievers (2021, methodology): “DPR could also leverage entity-aware embedding models like EaE (Févry et al., 2020) or LUKE (Ya-mada et al., 2020) to better recall long-tail entities.”
- Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, background): “It is possible that integrating entity supervision directly into LM pretraining (Févry et al., 2020) allows more efficient fact storage.”
- Knowledge-Aware Language Model Pretraining (2020, methodology): “To address this, previous research mainly grounds the language model to some forms of explicit semantic structures, for example, additional entity-oriented transformer layers [16, 18], or explicitly stored knowledge graph triples [19, 40, 41].”
- Retrieval-based Language Models and Applications (2023, methodology): “…et al., 2022b), 2) whether the retrieved components are latent and integrated into the intermediate layers of Transformers (de Jong et al., 2022; Févry et al., 2020; Borgeaud et al., 2022), or 3) distribution of tokens from the retrieved components and the LMs are interpolated (Khandelwal et…”
- Retrieval-Enhanced Machine Learning (2022, background): “Entities as Experts (EaE) [15] introduces an entity memory that can be accessed by the model and the retrieved representations of entities are combined with the input representation for entity linking, mention detection, and masked language modeling tasks.”