Entities as Experts: Sparse Memory Access with Entity Supervision

By Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, Tom Kwiatkowski. 2020

177 citations recorded by the source corpus.

We focus on the problem of capturing declarative knowledge in the learned parameters of a language model. We introduce a new model, Entities as Experts (EaE), that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EaE's entity representations are learned directly from text. These representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?". EaE outperforms a Transformer model with $30\times$ the parameters on this task. According to the Lama knowledge probes, EaE also contains more factual knowledge than a similar sized Bert. We show that associating parameters with specific entities means that EaE only needs to access a fraction of its parameters at inference time, and we show that the correct identification, and representation, of entities is essential to EaE's performance. We also argue that the discrete and independent entity representations in EaE make it more modular and interpretable than the Transformer architecture on which it is based.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Hannaneh Hajishirzi
  4. Yejin Choi
  5. Jianfeng Gao
  6. Percy Liang
  7. Caiming Xiong
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Hannaneh Hajishirzi
  2. Mohit Bansal
  3. Luc Van Gool
  4. Dacheng Tao
  5. Trevor Darrell
  6. Jianfeng Gao
  7. Xiaogang Wang
  8. Luke S. Zettlemoyer

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Akari Asai
  3. Danqi Chen
  4. Sebastian Riedel
  5. Wen-tau Yih
  6. Fabio Petroni
  7. Jinhyuk Lee
  8. Sewon Min

In Natural Language Processing

  1. Hannaneh Hajishirzi
  2. Mohit Bansal
  3. Luke S. Zettlemoyer
  4. Yejin Choi
  5. Sebastian Riedel
  6. Jianfeng Gao
  7. Wen-tau Yih
  8. Xiang Ren

Citations in the corpus