REALM: Retrieval-Augmented Language Model Pre-Training

By Kelvin Guu, Kenton C. T. Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. International Conference on Machine Learning, 2020

3,207 citations recorded by the source corpus.

Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search

View Paper: https://arxiv.org/abs/2002.08909

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Graham Neubig
  6. Yejin Choi
  7. Noah A. Smith
  8. Hannaneh Hajishirzi

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Sebastian Riedel
  3. Akari Asai
  4. Danqi Chen
  5. Wen-tau Yih
  6. Jason Aaron Edward Weston
  7. Luyu Gao
  8. Jamie Callan

In Information Retrieval & Search

  1. Jimmy J. Lin
  2. Jamie Callan
  3. Sebastian Riedel
  4. Luyu Gao
  5. Wen-tau Yih
  6. Hannaneh Hajishirzi
  7. Danqi Chen
  8. Gautier Izacard

Citations in the corpus