Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

By Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P. Parikh, Ali Farhadi, Hannaneh Hajishirzi. 2019

165 citations recorded by the source corpus.

Existing open-domain question answering (QA) models are not suitable for real-time usage because they need to process several long documents on-demand for every input query, which is computationally prohibitive. In this paper, we introduce query-agnostic indexable representations of document phrases that can drastically speed up open-domain QA. In particular, our dense-sparse phrase encoding effectively captures syntactic, semantic, and lexical information of the phrases and eliminates the pipeline filtering of context documents. Leveraging strategies for optimizing training and inference time, our model can be trained and deployed even in a single 4-GPU server. Moreover, by representing phrases as pointers to their start and end tokens, our model indexes phrases in the entire English Wikipedia (up to 60 billion phrases) using under 2TB. Our experiments on SQuAD-Open show that our model is on par with or more accurate than previous models with 6000x reduced computational cost, which translates into at least 68x faster end-to-end inference benchmark on CPUs. Code and demo are available at nlp.cs.washington.edu/denspi

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “retrieval method (Seo et al., 2019) and concurrent studies (Luan et al.”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Recently, Lee et al. (2019) and Seo et al. (2019) introduce fully trainable models that retrieve a few candidates directly from large-scale Wikipedia collections.”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “Other lines of research in open-domain QA include joint learning of retrieval and reader components (Lee et al., 2019) or direct phrase retrieval in a large collection of documents (Seo et al., 2019).”
Improving Wikipedia verifiability with AI (2022, background): “A document-level approach leads to worse results ( i.e. , ≈ 80 % precision at 15 % recall), mainly due to the impossibility of considering the whole document (given model architectural constraints on maximum input size).”
Learning Dense Representations of Phrases at Scale (2020, methodology): “Baselines For reading comprehension, we report scores of query-agnostic models including DenSPI (Seo et al., 2019), DenSPI + Sparc (Lee et al.”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “Prior work on hybrid sparse-dense models for QA has employed pipeline architectures (Seo et al., 2019; Karpukhin et al., 2020) where the first model’s candidates are rescored by the linear combination with the second model’s score; this is similar to our approach but has the disadvantage that the fi”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, neutral): “Other work has investigated incorporating information retrieval in language modeling and question answering (Chen et al., 2017; Fan et al., 2019; Seo et al., 2019; Guu et al., 2020), while we focus on dialogue applications and ﬂexibly incorporating knowledge from multiple, multi-modal sources.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “DenSPI [180] further combines dense and sparse vectors of phrases to capture both semantic and lexical information, so as to improve the performance of phrase-indexed question answering.”
Is Retriever Merely an Approximator of Reader? (2020, background): “However, for many recent question answering models, inner product space seems to lead to a similar or better model than L2 during training (Seo et al., 2019; Karpukhin et al., 2020), where one can still utilize asymmetric LSH (aLSH) (Shrivastava & Li, 2014) or k-means clustering.”
Look at the First Sentence: Position Bias in Question Answering (2020, background): “Also, instead of predicting positions, some models predict the n-grams as answers (Lee et al., 2016; Seo et al., 2019), generate answers in a vocabulary space (Raffel et al.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, methodology): “While previous work (Seo et al., 2019; Lee et al., 2020) relied on a combination of dense and sparse vectors, Lee et al. (2021) demonstrate that dense representations of phrases alone are sufﬁcient to close the performance gap with retriever-reader systems.”
Simple and Efficient ways to Improve REALM (2021, methodology): “Recently, dense retrieval methods (Seo et al., 2018, 2019; Das et al., 2019; Karpukhin et al., 2020) have improved over sparse retrievers like BM25 (Robertson and Zaragoza, 2009) and made training these systems end-to-end by leveraging approximate MIPS search (Shrivastava and Li, 2014).”
Distant Supervision for Multi-Stage Fine-Tuning in Retrieval-Based Question Answering (2020, methodology): “Furthermore, our techniques should be applicable to other neural models as well, including improvements to the basic BERTserini model architecture proposed by other researchers [1, 7, 8, 13, 20, 26].”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Bi-encoder QA has been used for efficient opendomain QA via phrase retrieval (Seo et al., 2018, 2019; Lee et al., 2020, 2021), but its lower accuracy compared to cross-encoder QA has previously been viewed as a drawback.”
Differentiable Open-Ended Commonsense Reasoning (2020, background): “To address the question, inspired by Seo et al. (2019) and Dhingra et al. (2020), we propose to construct an indexable representation of the corpus, and then reason over it via differentiable operations.”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, background): “This dense approach was recently extended by hybrid lexical-dense approaches which aims to combine the strengths of both approaches [17, 57, 42].”