REALM: Retrieval-Augmented Language Model Pre-Training
By Kelvin Guu, Kenton C. T. Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. International Conference on Machine Learning, 2020
3,207 citations recorded by the source corpus.
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search
View Paper: https://arxiv.org/abs/2002.08909
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Noah A. Smith
- Hannaneh Hajishirzi
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Akari Asai
- Danqi Chen
- Wen-tau Yih
- Jason Aaron Edward Weston
- Luyu Gao
- Jamie Callan
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Sebastian Riedel
- Luyu Gao
- Wen-tau Yih
- Hannaneh Hajishirzi
- Danqi Chen
- Gautier Izacard
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “An orthogonal approach to improving language models is to augment transformers with explicit retrieval mechanisms, as done by Borgeaud et al. (2021); Guu et al. (2020); Lewis et al. (2020).”
- PaLM: Scaling Language Modeling with Pathways (2022, background): “One area is retrieval models that aim to drastically reduce model sizes by embedding large amounts of text the model can have access to later (Guu et al., 2020; Borgeaud et al., 2021).”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Updating the document encoder during training is costly as it requires the document index to be periodically updated as REALM does during pre-training [18].”
- Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “…methods can use the labeled pairs to effectively generate a large number of examples, such as the in-batch random negative pairs (Yih et al., 2011; Henderson et al., 2017), or when there exist a big collection of positive and negative pairs (Huang et al., 2013), dense representation can…”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “…31.7 - - 56.5 Graph Retriever (Min et al., 2019b) 34.7 55.8 - - Hard EM (Min et al., 2019a) 28.8 50.9 - - ORQA (Lee et al., 2019) 31.3 45.1 - 20.2 REALM (Guu et al., 2020) 38.2 - - DPR (Karpukhin et al., 2020) 41.5 57.9 - 36.7 SpanSeqGen (Min et al., 2020) 42.5 - - - RAG (Lewis et al., 2020) 44”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…information during pretraining, including various forms of metadata (Keskar et al., 2019), HTML tags (Aghajanyan et al., 2021), Wikipedia markup (Schick et al., 2022), or related texts obtained from an information retrieval system (Guu et al., 2020; Borgeaud et al., 2021; Izacard et al., 2022).”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “…As described previously, we observed significant performance gains with additional pre-training using “salient span masking” (SSM) on the Wikipedia sentence dataset from Guu et al. (2020) but not when using the standard “span corruption” (SC) from Raffel et al. (2019) on longer Wikipedia articles.”
- LaMDA: Language Models for Dialog Applications (2022, methodology): “Given these differences, we note that the range of existing approaches to this problem include the RNNLM [34], RAG [35], REALM [36], and FiD [37] architectures.”
- Improving language models by retrieving from trillions of tokens (2021, background): “This potential issue is specifically addressed by (Guu et al., 2020), which trains the retrieval system end-to-end to maximize the final training cross-entropy.”
- Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “Guu et al. (2020) proposed to augment language model pre-training with a retriever module, which is trained using the masked language modeling objective.”
- Big Bird: Transformers for Longer Sequences (2020, methodology): “Most prominently, SpanBERT [42], ORQA [54], REALM [34], RAG [57] have achieved strong performance for different tasks.”
- Emergent Abilities of Large Language Models (2022, neutral): “…inputs (Graves, 2016; Dehghani et al., 2018), using more localized learning strategies than backpropagation through all weights in a neural network (Jaderberg et al., 2017), and augmenting models with external memory (Guu et al., 2020; Borgeaud et al., 2021; Wu et al., 2022b, inter alia ).”
- Augmented Language Models: a Survey (2023, methodology): “REALM (Guu et al., 2020) was the first method to jointly train end-to-end a retrieval system with an encoder LM.”
- WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “Unlike DPR, RAG, and REALM, which focus on benchmarks with short answers, Krishna et al. [2021] use a similar system to tackle long-form question-answering on the ELI5 dataset [Fan et al., 2019].”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, neutral): “For example, REALM [11] jointly trains the reader and retriever, where both components are transformers.”
- A Review of Sparse Expert Models in Deep Learning (2022, background): “Retrieval mechanisms effectively expand the capacity of models by allowing them to dynamically access information beyond the current context or what is stored in the parameters (Khandelwal et al., 2019; Guu et al., 2020; Borgeaud et al., 2022).”