Simple Entity-Centric Questions Challenge Dense Retrievers

By Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, Danqi Chen. 2021

213 citations recorded by the source corpus.

Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples. However, in this paper, we demonstrate current dense models are not yet the holy grail of retrieval. We first construct EntityQuestions, a set of simple, entity-rich questions based on facts from Wikidata (e.g., “Where was Arve Furset born?”), and observe that dense retrievers drastically under-perform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions towards addressing this critical problem. First, we demonstrate that data augmentation is unable to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using specialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Question Answering & Reading Comprehension

Citations in the corpus

When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, methodology): “Overall, Contriever-guided LMs outperform BM25-based ones on P OP QA, while the BM25-based models perform better on the least popular entities, consistent with the ﬁndings from Sciavolino et al. (2021).”
Boosted Dense Retriever (2021, methodology): “We also evaluate NQ-trained models on EntityQuestions (Sciavolino et al., 2021), a dataset of simple entity-centric questions which has been recently shown to challenge dense retrievers.”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, background): “, 2021; Izacard & Grave, 2020) and on performance analysis of the dense retrievals, for instance, when dealing with named entities (Sciavolino et al., 2021; Liu et al., 2021).”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “In a further stage of generator training, the model acknowledges the critical role of entity semantics in the representation learning of textual data for retrieval, as highlighted by [ Sciavolino et al. , 2021, Zhang et al. , 2019 ] .”
Improving Passage Retrieval with Zero-Shot Question Generation (2022, background): “Previous work on this dataset has shown that dense retrievers struggle to retrieve relevant passages while sparse approaches like BM25 are more successful (Sciavolino et al., 2021).”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Overall, Contriever-guided LMs show stronger performance than BM25-based ones, while the BM25-based models show their superiority on extremely less popular entities, consistent with the ﬁndings from Sciavolino et al. (2021).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “8M – – 3,213,835 Wikipedia EntityQuestions [52] 176,560 186,367 22,068 22,075 – Web MS MARCO v2 [53] 277,144 284,212 8,184 – 138,364,198 Web DuReaderretrieval [54] 97,343 86,395 2,000 8,948 8,096,668”
Effects of Model Choice, Corpus Context, and Post Hoc Correction on Layer-Level Embedding Degradation in Clinical Document Retrieval: Experimental Study (2026, unknown): “Dense retrieval architectures pioneered by Karpukhin et al [22] have been widely adopted for clinical and biomedical retrieval, with their failure modes subsequently characterized in BEIR-style evaluations [23].”
Improving Code Localization with Repository Memory (2025, unknown): “Similar phenomena are also observed in prior work, e.g., Sciavolino et al. (2021) finds that dense retrievers could drastically underperform sparse methods in entity-centric question-answering.”