RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking

By Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, Ji-rong Wen. Conference on Empirical Methods in Natural Language Processing, 2021

307 citations recorded by the source corpus.

In various natural language processing tasks, passage retrieval and passage re-ranking are two key procedures in finding and ranking relevant information. Since both the two procedures contribute to the final performance, it is important to jointly optimize them in order to achieve mutual improvement. In this paper, we propose a novel joint training approach for dense passage retrieval and passage reranking. A major contribution is that we introduce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other’s relevance information. We also propose a hybrid data augmentation strategy to construct diverse training instances for listwise training approach. Extensive experiments show the effectiveness of our approach on both MSMARCO and Natural Questions datasets. Our code is available at https://github.com/PaddlePaddle/RocketQA.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1cef73714d8ad89e442a9635fcd3061c61067638

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Building on the practices of training state-of-the-art dense retrievers [50, 58], we use mined hard negatives and knowledge distillation from a cross-encoder (CE) teacher model for the MS-MARCO and NQ datasets.”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, methodology): “While ColBERT outperforms single-vector systems like RepBERT, ANCE, and even TAS-B, improvements in supervision such as distillation from cross-encoders (see §2) enable systems like SPLADEv2, PAIR, and RocketQAv2 to achieve higher quality than vanilla ColBERT.”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “Distilling interaction based models into representation based models has been shown effective in closing the gap between the two (Hofstätter et al., 2020; Ren et al., 2021a; Lin et al., 2021; Ren et al., 2021b; Reddi et al., 2021; Zhang et al., 2022).”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, unknown): “In addition, the progress of contrastive learning is another critical factor, especially the improvement of negative sampling (Xiong et al., 2020; Qu et al., 2021) and the exploitation of knowledge distillation (Hofst¨atter et al., 2021; Ren et al., 2021; Zhang et al., 2021a).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Thakur et al. [72] conduct an analysis experiment by training two BERTbased models (an identical parameter configuration on MS MARCO dataset) with cosine similarity and inner product, respectively.”
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, background): “In our cross-encoder ranker, all the query tokens can interact with all the document tokens thus it has more substantial relevance modeling power than other ranker architectures, such as dual- or biencoders [29, 31, 40, 54].”