Dense Passage Retrieval for Open-Domain Question Answering
By Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih. Conference on Empirical Methods in Natural Language Processing, 2020
6,334 citations recorded by the source corpus.
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search
View Paper: https://arxiv.org/abs/2004.04906
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Hannaneh Hajishirzi
- Caiming Xiong
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Danqi Chen
- Akari Asai
- Jason Aaron Edward Weston
- Wen-tau Yih
- Luyu Gao
- Jamie Callan
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Wen-tau Yih
- Luyu Gao
- Sebastian Riedel
- Hannaneh Hajishirzi
- Sebastian Hofstätter
- Gautier Izacard
Citations in the corpus
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Hybrid models that combine parametric memory with non-parametric (i.e., retrieval-based) memories [18, 22, 42] can address some of these issues because knowledge can be directly revised and expanded, and its access can be inspected and interpreted.”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “2 In DPR, passages and questions are represented as dense vector representations, computed using two BERT networks.”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Recently, generating the answer as free-form text, using a seq2seq model conditioned on retrieved documents have become prevalent (Lewis et al., 2020; Izacard & Grave, 2020; Min et al., 2020).”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “For NatQA, following (Min et al., 2020), we use the DPR retrieval engine (Karpukhin et al., 2020) to augment each question with additional paragraphs.”
- Improving language models by retrieving from trillions of tokens (2021, methodology): “We use the version 4 provided by Izacard and Grave (2021) which is augmented with the retrieved passages from (Karpukhin et al., 2020).”
- Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “A critical component of the iterative training procedure is the initialization of the support passages D 0 q asso-1 Model NQ TriviaQA dev. test dev. test DPR (Karpukhin et al., 2020) - 41.5 - 57.9 RAG (Lewis et al., ciated with each question q .”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Various methods have been proposed for training retrievers and readers [16].”
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “All DR baselines use the same BERT-Siamese (base) model as used in various parallel research [9, 12, 13, 15].”
- Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “5 EM in (Karpukhin et al., 2020)) even with a much smaller index dimension.”
- AmbigQA: Answering Ambiguous Open-domain Questions (2020, methodology): “We use the exact same setup and hyperparameters for any process that we follow Karpukhin et al. (2020).”
- Re2G: Retrieve, Rerank, Generate (2022, methodology): “DPR Stage 1 training is the same training used by Karpukhin et al. [2020].”
- Improving Wikipedia verifiability with AI (2022, methodology): “The context and passage encoders are trained such that the context and passage vectors of existing Wikipedia citation and evidence pairs are maximally similar (Karpukhin et al., 2020).”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “…for query expansion in traditional search engines (Mao et al., 2021), but these solutions don’t exploit the full potential of autoregressive architecture, such as word order sensitivity and conditional probability modeling, and still lag behind vector-based approaches (Karpukhin et al., 2020).”
- The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “We consider two baseline retrieval models: BM25 (Robertson, 2009) and DPR (Karpukhin et al., 2020).”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently (Huang et al., 2013; Karpukhin et al., 2020).”