PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval

By Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, Ji-rong Wen. 2021

103 citations recorded by the source corpus.

Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

You Only Need One Model for Open-domain Question Answering (2021, methodology): “We also compare our pre-training to that of the additional data augmentation (Mao et al., 2021a; Ren et al., 2021; Oguz et al., 2021).”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “Distilling interaction based models into representation based models has been shown effective in closing the gap between the two (Hofstätter et al., 2020; Ren et al., 2021a; Lin et al., 2021; Ren et al., 2021b; Reddi et al., 2021; Zhang et al., 2022).”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “The dense retrievers include DPR (Karpukhin et al., 2020), DPR-E, ANCE (Xiong et al., 2020a), ME-BERT (Luan et al., 2021), NPRINC (Lu et al., 2020), ColBERT (Khattab and Zaharia, 2020) RocketQA (Qu et al., 2021), TAS-Balanced (Hof-stätter et al., 2021), ADORE+STAR (Zhan et al., 2021) and PAIR (Ren e”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, methodology): “…directions: (1) distillation of more expressive architectures (Hofstätter et al., 2020; Lin et al., 2020) including explicit denoising (Qu et al., 2021; Ren et al., 2021b), (2) hard negative sampling (Xiong et al., 2020; Zhan et al., 2020a), and (3) improved pretraining (Gao and Callan, 2021).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Furthermore, we implement and release a code repertory for a number of dense retrieval models based on PaddlePaddle3 (including RocketQA [36], PAIR [37] and RocketQAv2 [38]), and provide flexible interfaces to use or retrain dense retrievers.”
Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, background): “…) , ..., ( q n , d n ) } , the authors propose an improved contrastive loss (icl) can be viewed as a combination of loss variants proposed by [66], [67], [68]: where Z = j e s ( q i ,d j ) /τ + j ̸ = i e s ( q i ,q j ) /τ + j e s ( q j ,d i ) /τ + j ̸ = i e s ( d j ,d i ) /τ (3) The cosine…”