RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering

By Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, Haifeng Wang. 2020

749 citations recorded by the source corpus.

In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding answers. Typically, the dual-encoder architecture is adopted to learn dense representations of questions and passages for semantic matching. However, it is difficult to effectively train a dual-encoder due to the challenges including the discrepancy between training and inference, the existence of unlabeled positives and limited training data. To address these challenges, we propose an optimized training approach, called RocketQA, to improving dense passage retrieval. We make three major technical contributions in RocketQA, namely cross-batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions. We also conduct extensive experiments to examine the effectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

You Only Need One Model for Open-domain Question Answering (2021, methodology): “This approach was also applied in Rock-etQA (Qu et al., 2021).”
Learning to Retrieve Passages without Supervision (2021, background): “Dense representations are the state-of-the-art models for retrieval in open-domain question answering (Lee et al., 2019; Karpukhin et al., 2020; Qu et al., 2021).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “While there are aspects of text ranking with pretrained trans-formers that are well understood, many promising directions await further exploration.”
Learning To Retrieve Prompts for In-Context Learning (2021, background): “Retrieval Research on training dense retrievers has skyrocketed recently, propelled by interest in open-domain question answering (Chen et al., 2017; Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Khattab and Zaharia, 2020; Qu et al., 2021).”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, methodology): “We advance the best published results (Qu et al., 2021) by +1.7 points on both top-20 and top-100 accuracy.”
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021, methodology): “Recent progress has been made towards improving evidence retrieval through both sparse vector models like TF-IDF or BM25 (Chen et al., 2017; Min et al., 2019), and dense vector models based on BERT (Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Qu et al., 2021).”
Boosting Search Engines with Interactive Agents (2021, neutral): “…has driven signiﬁcant innovation in IR and QA research by allowing direct comparison of many diﬀerent approaches in a fast moving landscape; e.g. from ORQA (Lee et al., 2019) to closed book QA (Roberts et al., 2020) to RAG (Lewis et al., 2020b; Qu et al., 2021), DPR (Karpukhin et al., 2020) etc.”
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval (2021, methodology): “Following RocketQA (Qu et al., 2020), we adopt the cross-encoder architecture to”
Optimizing Dense Retrieval Model Training with Hard Negatives (2021, methodology): “We start with introducing the widely-used random negative sampling method [7, 12, 29].”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, methodology): “coming from BM25 sampling), and a set of negative documents in the batch (positive documents from other queries) {d− i, j } j , we consider the ranking loss from [8], which can be interpreted as the maximization of the probability of the document d+ i being relevant among the documents d+ i , d− i a”
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval (2021, methodology): “…(ours) 0.312 0.941 0.637 0.711 ANCE [29] 0.330 0.959 0.648 - TCT-ColBERT [16] 0.359 0.970 0.719 0.760 TAS-B [11] 0.347 0.978 0.717 0.843 RocketQA [24] 0.370 0.979 - - Sparse retrieval BM25 0.184 0.853 0.506 0.745 DeepCT [4] 0.243 0.913 0.551 0.756 doc2query-T5 [20] 0.277 0.947 0.642 0.827…”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, methodology): “The recent RocketQA system (Qu et al., 2021) signiﬁcantly improves the performance of a dense retriever by designing an optimized ﬁne-tuning pipeline that includes i) denoising hard negatives, which corrects mislabeling, and ii) large batch training.”
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup (2021, background): “Supervised contrastive learning (Khosla et al., 2020) has also been shown effective in training dense retrievers (Karpukhin et al., 2020; Qu et al., 2020).”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “The community produces mounting evidence that knowledge distillation is essential for effective dense retrieval training [11, 14, 24].”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, positive): “…DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2021), but also encompassing many other techniques (Gao et al., 2021b; Hofstätter et al., 2020; Qu et al., 2021; Hofstätter et al., 2021; Lin Table 1: Our conceptual framework for organizing recent developments in information retrieval. et al.,…”
SGPT: GPT Sentence Embeddings for Semantic Search (2022, neutral): “To balance the trade-offs, multi-stage architectures have been proposed [30, 34, 21].”