Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
By Gautier Izacard, Edouard Grave. Conference of the European Chapter of the Association for Computational Linguistics, 2020
1,909 citations recorded by the source corpus.
Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that sequence-to-sequence models offers a flexible framework to efficiently aggregate and combine evidence from multiple passages.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing
View Paper: https://arxiv.org/abs/2007.01282
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Hannaneh Hajishirzi
- Graham Neubig
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Jianfeng Gao
- Andrew Zisserman
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Akari Asai
- Danqi Chen
- Luyu Gao
- Jamie Callan
- Jason Aaron Edward Weston
- Wen-tau Yih
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Graham Neubig
- Jason Aaron Edward Weston
- Percy Liang
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Recently, generating the answer as free-form text, using a seq2seq model conditioned on retrieved documents have become prevalent (Lewis et al., 2020; Izacard & Grave, 2020; Min et al., 2020).”
- LaMDA: Language Models for Dialog Applications (2022, methodology): “Given these differences, we note that the range of existing approaches to this problem include the RNNLM [34], RAG [35], REALM [36], and FiD [37] architectures.”
- Improving language models by retrieving from trillions of tokens (2021, methodology): “We use the version 4 provided by Izacard and Grave (2021) which is augmented with the retrieved passages from (Karpukhin et al., 2020).”
- Big Bird: Transformers for Longer Sequences (2020, methodology): “Also, high performing Fusion-in-Decoder [40] model for TriviaQA can be considered to be”
- Augmented Language Models: a Survey (2023, background): “Izacard and Grave (2020) introduced a Efficient large scale retrieval.”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2020), Retrieval-based (de Masson d'Autume et al., 2019; Izacard and Grave, 2021; Hossain et al., 2020); Summarization (Huang et al.”
- Re2G: Retrieve, Rerank, Generate (2022, methodology): “The reader component is the Fusion-in-Decoder [Izacard and Grave, 2021] model, where retrieved documents are encoded independently, then their encoded representations are concatenated for the decoder.”
- Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, methodology): “We train an FiD model (Izacard & Grave, 2020) using T5 v1.”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “The former retrieves relevant context, usually in the form of short passages, which the latter then examines to produce answers (Chen et al., 2017; Lewis et al., 2020; Izacard and Grave, 2021).”
- Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “(Izacard and Grave, 2021b) which may result in incorporating parts of multiple documents into one factually incorrect response.”
- Internet-augmented language models through few-shot prompting for open-domain question answering (2022, methodology): “When applied to question answering tasks [7, 8, 18], they surpass performance of parametric-only models – they are able to efficiently handle an increasing number of retrieved passages and ground their predictions into additional information, thus reducing hallucinations and improving factuality.”
- The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “In this work, we use an abstractive Fusion-in-Decoder (F I D) reader from Izacard and Grave (2020) — an encoder-decoder architecture, where each context document is concatenated with the input and embedded by the encoder.”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard & Grave, 2020a), or the relevance score from a cross encoder (Yang & Seo, 2020).”
- You Only Need One Model for Open-domain Question Answering (2021, methodology): “Data for the Wikipedia dump, Natural Questions, and TriviaQA can also be downloaded from FiD’s github using this script.”
- Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, methodology): “Indeed, recent work (Sachan et al., 2021) illustrates by using Fusion-in-Decoder during inference but foregoing the de-coder’s attention weights and using an equivalent version of the M ARGINALIZED L OSS for training the retriever.”
- R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “Contrary to belief based on the results on different datasets (Yang et al., 2019; Wang et al., 2019b; Izacard and Grave, 2021), we found the extractive reader can also benefit from larger input sizes, both in training and test time.”
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “Our reader is based on the Fusion-in-Decoder (FiD; Izacard and Grave, 2021b) model, which is built on top of T5 (Raffel et al.”