You Only Need One Model for Open-domain Question Answering
By Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher D. Manning, Kyoung-Gu Woo. 2021
31 citations recorded by the source corpus.
Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank passages with a separate reranker model and generate an answer using another reader model. Despite performing related tasks, the models have separate parameters and are weakly-coupled during training. We propose casting the retriever and the reranker as internal passage-wise attention mechanisms applied sequentially within the transformer architecture and feeding computed representations to the reader, with the hidden representations progressively refined at each stage. This allows us to use a single question answering model trained end-to-end, which is a more efficient use of model capacity and also leads to better gradient flow. We present a pre-training method to effectively train this architecture and evaluate our model on the Natural Questions and TriviaQA open datasets. For a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores.
Subfields: Information Retrieval & Search, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Retrieval & Search
- Luyu Gao
- Sebastian Riedel
- Fabio Petroni
- Jamie Callan
- Patrick Lewis
- Gautier Izacard
- Edouard Grave
- Timo Schick
In Natural Language Processing
- Sebastian Riedel
- Graham Neubig
- Patrick Lewis
- Edouard Grave
- Timo Schick
- Fabio Petroni
- Luke S. Zettlemoyer
- Luyu Gao
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Guu et al. (2020) also explore a query-side finetuning at finetuning time to avoid index refreshes, which is also explored in the context of phrase-based retrieval by Lee et al. (2021b).”
- Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, methodology): “We compare retrieval performance with lexical models BM25 (Robertson and Zaragoza, 2009), passage-level dense retrievers DPR, ANCE, coCon-denser, FiD-KD, YONO (with and without retrieval pretraining) (Karpukhin et al., 2020; Oguz et al., 2021; Xiong et al., 2021; Gao and Callan, 2022; Izacard and Gr”
- Active Retrieval Augmented Generation (2023, unknown): “…answer conditioning on the retrieved documents (Chen et al., 2017; Guu et al., 2020; Lewis et al., 2020; Izacard and Grave, 2021; Sachan et al., 2021; Lee et al., 2021; Jiang et al., 2022; Izacard et al., 2022; Nakano et al., 2021; Qian et al., 2023; Lazaridou et al., 2022; Shi et al., 2023).”