Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?

By Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stanislav Peshterliev, Yashar Mehdad, Sonal Gupta, Wen-tau Yih. 2021

87 citations recorded by the source corpus.

Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query and to generalize to out-of-domain data. It has been argued that this is an inherent limitation of dense models. We rebut this claim by introducing the Salient Phrase Aware Retriever (SPAR), a dense retriever with the lexical matching capacity of a sparse model. We show that a dense Lexical Model {\Lambda} can be trained to imitate a sparse one, and SPAR is built by augmenting a standard dense retriever with {\Lambda}. Empirically, SPAR shows superior performance on a range of tasks including five question answering datasets, MS MARCO passage retrieval, as well as the EntityQuestions and BEIR benchmarks for out-of-domain evaluation, exceeding the performance of state-of-the-art dense and sparse retrievers. The code and models of SPAR are available at: https://github.com/facebookresearch/dpr-scale/tree/main/spar

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “We train DPR WEB by ﬁnetuning a PAQ-based (Lewis et al., 2021) bi-encoder check-point (O˘guz et al., 2021) for 40 epochs on 16 GPUs.”
Task-aware Retrieval with Instructions (2022, background): “, 2016) and then performs a transfer to new datasets (Izacard et al., 2022; Khattab and Zaharia, 2020; Santhanam et al., 2022; Nogueira et al., 2020; Chen et al., 2022).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “As an alternative approach, knowledge distillation is commonly adopted for tackling the scarcity of training data, which utilizes a more capable teacher model to improve the student model [102], [123], [126] in zero-shot retrieval scenarios.”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “…generalization, such as using query generation for data augmentation (Ma et al., 2021; Wang et al., 2022), using contrastive learning for better pre-training (Izacard et al., 2022a), using knowledge distillation (Chen et al., 2021; Wang et al., 2022) and scaling the model size (Ni et al., 2021).”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “SPAR [11] trains a dense retriever by treating BM25 as a teacher model.”
Boosted Dense Retriever (2021, methodology): “We are interested to test whether our method could be beneﬁcial for out-of-domain trans-Method EntityQuestions R@20 R@100 BM25 (Chen et al., 2021) 71.2 79.7 DPR (Chen et al., 2021) 49 fer as well.”
DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Inspired by methods such as SPAR (Chen et al., 2022) and DRAGON (Lin et al., 2023), we begin with the corpus C .”