End-to-End Training of Neural Retrievers for Open-Domain Question Answering

By Devendra Singh Sachan, M. Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro. 2021

110 citations recorded by the source corpus.

Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsupervised approaches. However, it remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2+ points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We next explore two approaches for end-to-end training of the reader and retriever components in OpenQA models, which differ in the manner the reader ingests the retrieved documents. Our experiments demonstrate the effectiveness of these approaches as we obtain state-of-the-art results. On the Natural Questions dataset, we obtain a top-20 retrieval accuracy of 84%, an improvement of 5 points over the recent DPR model. We also achieve good results on answer extraction, outperforming recent models like REALM and RAG by 3+ points.

Subfields: Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “On QA datasets, we compare to dense retrievers trained with ICT and the Masked Salient Spans from Sachan et al. (2021).”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “On TriviaQA, ours performs slightly worse than the state-of-the-art models, FiD-KD (Izacard and Grave, 2021a) and E2NR (Sachan et al., 2021), which use passage labels during training or 4x more parameters.”
Learning to Retrieve Passages without Supervision (2021, methodology): “Sachan et al. (2021a) trained their own ICT model, which shows stronger performance than Lee et al. (2019).”
Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, methodology): “Indeed, recent work (Sachan et al., 2021) illustrates by using Fusion-in-Decoder during inference but foregoing the de-coder’s attention weights and using an equivalent version of the M ARGINALIZED L OSS for training the retriever.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “Similar to previous methods like REALM and RAG, Individual Top-K objective function is also defined over multiple retrieved documents but is better optimized than them.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, positive): “Concurrent work (Sachan et al., 2021) combined ICT pre-training with masked-salient-span pre-training, as well as an end-to-end ﬁne-tuning using a T5-large model, obtaining results comparable or slightly better than what’s presented here.”
Boosted Dense Retriever (2021, methodology): “Recently, it has become common practice to run a number of rounds of dense retrieval training to bootstrap hard negatives (Xiong et al., 2020; Qu et al., 2021; O ˘ guz et al., 2021; Sachan et al., 2021, inter alia.).”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “Furthermore, several studies attempted to jointly train the retriever and the reader for Open-domain Question Answering (Guu et al., 2020; Sachan et al., 2021; Karpukhin et al., 2020).”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, result): “While showing promising results as pre-training (Chang et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when used as a zero-shot retriever.”
Large Dual Encoders Are Generalizable Retrievers (2021, background): “Other techniques, such as negative mining (Xiong et al., 2020; Lu et al., 2021; Sachan et al., 2021) and having large training batch sizes (Qu et al., 2021), are also highly effective.”
Text and Code Embeddings by Contrastive Pre-Training (2022, methodology): “Previous work on training embedding methods for search typically requires ﬁne-tuning on a particular text search dataset (Karpukhin et al., 2020a; Sachan et al., 2021; Qu et al., 2021).”
Improving Passage Retrieval with Zero-Shot Question Generation (2022, methodology): “2B Joint Top-K (large) (Sachan et al., 2021a) 50 - - - 68.”
Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering (2022, methodology): “Motivated by recent end2end retrieval augmented mechanisms (Guu et al., 2020; Sachan et al., 2021; Singh et al., 2021), we first propose a method to finetune the RAG model with its neural retriever and update its knowledge encodings asynchronously during training.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “[108] present an end-to-end training method to jointly optimize the retriever and the reader”
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2021, methodology): “We also compare with methods (4) with additional largescale pre-training: REALM (Guu et al., 2020), RAG (Lewis et al., 2020) and Joint Top-K (Sachan et al., 2021).”