Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

By Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Michael Lewis, Wen-tau Yih, Tim Rocktaschel and 2 more. Neural Information Processing Systems, 2020

16,110 citations recorded by the source corpus.

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Retrieval-Augmented Generation, Information Retrieval & Search

View Paper: https://arxiv.org/abs/2005.11401

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Hannaneh Hajishirzi
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Information Systems

  1. Jamie Callan
  2. Jimmy J. Lin
  3. Hannaneh Hajishirzi
  4. Sebastian Riedel
  5. Zhuyun Dai
  6. Luke S. Zettlemoyer
  7. Percy Liang
  8. Wen-tau Yih

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Sebastian Riedel
  3. Akari Asai
  4. Danqi Chen
  5. Jason Aaron Edward Weston
  6. Wen-tau Yih
  7. Luyu Gao
  8. Jamie Callan

In Information Retrieval & Search

  1. Jamie Callan
  2. Jimmy J. Lin
  3. Wen-tau Yih
  4. Sebastian Riedel
  5. Hannaneh Hajishirzi
  6. Luyu Gao
  7. Danqi Chen
  8. Gautier Izacard

Citations in the corpus