Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
By Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Michael Lewis, Wen-tau Yih, Tim Rocktaschel and 2 more. Neural Information Processing Systems, 2020
16,110 citations recorded by the source corpus.
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Retrieval-Augmented Generation, Information Retrieval & Search
View Paper: https://arxiv.org/abs/2005.11401
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Hannaneh Hajishirzi
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Information Systems
- Jamie Callan
- Jimmy J. Lin
- Hannaneh Hajishirzi
- Sebastian Riedel
- Zhuyun Dai
- Luke S. Zettlemoyer
- Percy Liang
- Wen-tau Yih
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Akari Asai
- Danqi Chen
- Jason Aaron Edward Weston
- Wen-tau Yih
- Luyu Gao
- Jamie Callan
In Information Retrieval & Search
- Jamie Callan
- Jimmy J. Lin
- Wen-tau Yih
- Sebastian Riedel
- Hannaneh Hajishirzi
- Luyu Gao
- Danqi Chen
- Gautier Izacard
Citations in the corpus
- Language Models are Few-Shot Learners (2020, background): “The one-shot result improves by 3.7% and matches the SOTA for an open-domain QA system which not only fine-tunes but also makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [LPP + 20].”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Recently, generating the answer as free-form text, using a seq2seq model conditioned on retrieved documents have become prevalent (Lewis et al., 2020; Izacard & Grave, 2020; Min et al., 2020).”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, background): “…find more examples do not improve performance. a HotpotQA EM is 27.1, 28.9, 33.8 for Standard, CoT, CoT-SC in Wang et al. (2022b). b (Zhu et al., 2021; Lewis et al., 2020) search reformulation (“maybe I can search/look up x instead”), and synthesize the final answer (“...so the answer is…”
- LaMDA: Language Models for Dialog Applications (2022, methodology): “Given these differences, we note that the range of existing approaches to this problem include the RNNLM [34], RAG [35], REALM [36], and FiD [37] architectures.”
- Improving language models by retrieving from trillions of tokens (2021, background): “RAG (Lewis et al., 2020) and FiD (Izacard and Grave, 2021) (Sachan et al., 2021) extends FiD by using an expectation-maximization algorithm to train the retriever end-to-end and achieves state of the art results compared to similarly sized models.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “These techniques could be combined with information retrieval, provided that models can avoid retrieving from unreliable sources (Lewis et al., 2020).”
- Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Lewis et al. (2020b) further train the retriever obtained in Karpukhin et al. (2020) by backpropagating to the retriever the error between the generated output and the gold answer.”
- Big Bird: Transformers for Longer Sequences (2020, methodology): “Most prominently, SpanBERT [42], ORQA [54], REALM [34], RAG [57] have achieved strong performance for different tasks.”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “We aim to be fully accountable for the development lifecycle of OPT-175B, and only through increasing transparency around LLM development can we start understanding the limitations and risks of LLMs before broader deployment occurs.”
- Augmented Language Models: a Survey (2023, methodology): “…by adding the retrieved documents to the current context (Chen et al., 2017; Clark and Gardner, 2017; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020; Zhong et al., 2022; Borgeaud et al., 2022; Izacard et al., 2022; Shi et al., 2023).”
- Galactica: A Large Language Model for Science (2022, methodology): “Examples of such models include RAG, RETRO and Atlas (Lewis et al., 2020b; Borgeaud et al., 2021; Izacard et al., 2022).”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “In contrast, Lewis et al. (2020); de Masson d'Autume et al. (2019); Izacard and Grave (2021); Hossain et al. (2020) use unstructured text as knowledge.”
- WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “Previous work on question-answering such as REALM [Guu et al., 2020] and RAG [Lewis et al., 2020a] has focused on improving document retrieval for a given query.”